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contains a model for integrating program planning, implementation, 
and evaluation activities. Chapter 3 addresses initial assessment, 
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PREFACE 



// we don 't change directions, we will end up where we are headed. 

—Chinese proverb 

Assessment and evaluation strategies help staff members and learners identify 
what direction their family literacy project is taking. The strategies also 
)rovide information that can be used for staff and learners who want to change 
course to achie different results. This handbook was de ~loped to assist 
staff members to design and use alternative approaches to assessment and 
evaluation. Most family liieracy projects use standardized as well as alterna- 
tive measures. By using a variety of approaches, staff can obtain in-depth 
information about the learners' backgrounds, the progress the learners are 
making, and the overall effectiveness of the project. The handbook does not 
contain all the information needed to design a comprehensive evaluation of a 
project; rather, it is intended as a supplemental resource for creating and using 
alternative approaches. Because family English literacy projects have a great 
deal in common with oiher family literacy projects, the information should be 
helpful to those working to improve assessment, evaluation, and instruction in 
projects serving language-minority families. These projects might be funded 
through a variety of sources. 

development of the handbook began in December 1990 at a coordina- 
tion meeting of fourteen Family English Literacy Projects (FELP) in Califor- 
nia funded by Title VII of the Elementary and Secondary Education Act 
(ESEA). The needs assessment conducted then indicated that staff considered 
assessment and evaluation as among the most critical challenges they faced. 
FELP staff emphasized the need for procedures and instruments lailored to the 
learners, curriculum, schedule, and other key characteristics of their projects. 
Although standardized tests provided a partial view of what was happening in 
their projects, they needed additional strategies for assessing language profi- 
ciency as well as hard-to- measure outcomes such as self-confidence and 
literacy awareness. 

As they wrote their chapters, the authors worked closely with family 
literacy staff to create a handbook that would meet their needs for alternative 
assessment. The authors met frequently wilh representatives of family literacy 
projects and obtained feedback on drafts of their chapters. In general, family 
literacy staff stated that the handbook should do the following: 

1. Demystify assessment and evaluation. Staff members of family 
literacy projects can benefit from direction in assessment and evaluation in 
consultation with a professional evaluator. 

2. Present concrete suggestions lor designing and using alternative 
assessment instruments and procedures. Staff want information that is imme- 
diately useful and sample instruments that can be adapted for their projects. 

3. Focus on a limited number of assessment approaches. Rather than 
presenting general information about many different alternative measures, the 

rhors concentrated on four alternative approaches: surveys, interviews, 
observation measures, and performance samples. 
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The chapters are organized according to the sequence in which 
alternative approaches are likely to be considered, designed, and used in a 
family literacy project. Chapter 1 contains an introduction to family literacy 
and presents some of the special evaluation issues related to implementing 
a family literacy project. Next, Chapter 11 presents a model for integrating 
program planning, implementation, and evaluation. Chapters 111, IV, and V 
illustrate the three phases of assessment and evaluation; (1 ) determining 
learners* needs and establishing baseline data; (2) documenting learners* 
progress toward meeting project objectives; and (3) collecting, analyzing, 
using, and reporting data obtained with alternative approaches. Chapter VI 
provides an overview of the development of the handbook and a description 
of the revisions that were made based on comments received about the 
field-test edition. 

The examples of alternative assessment instruments presented in the 
handbook are not intended as ready-to-use assessment tools; rather, the 
authors understand that staff members will modify these tools or create 
their own instruments and approaches based on the unique characteristics 
of the learners, staff, and program design. As the instruments and other 
information in the volume are adapted to the specific needs of individual 
projects, assessment and evaluation of family literacy projects will become 
more beneficial to learners and staff. 

Mary 7. Mahony 

Office of Bilingual Education and Minority Languages Affairs 
U.S. Department of Education 
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The Family 

It is a fragile place 

In constant search of balance. 

Where children look for love and safely 

To explore life's wonders. 

Where parents snuggle 

With holding on and letting go. 

It flourishes when its members 

Become everything they are and want to be. 

When each one's needs are met 

And each supportsS the others to meet theirs. 



Daniel D. Holt 
Sacramento, California 
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CHAPTER I 

ALTERNATIVE 
APPROACHES TO 
ASSESSMENT AND 
EVALUATION 



Among the many responsibilities of staff members of a family literacy 
project* are the tasks of assessing and evaluating their project. Such 
responsibility involves developing a comprehensive evaluation design to 
determine the degree to which project goals and objectives are met. The 
purpose of this handbook is to assist staff members with implementing a 
key part of effective evaluation — that is, designing and implementing 
alternative approaches to assessment. By integrating alternative assessment 
approaches with standardized assessment measures, staff can obtain more 
accurate and complete information to improve their project. Much of the 
information contained in this and other chapters of the handbook is based 
on the experiences of staff members and learners in projects funded under 
ESEA Title VII Family English Literacy Programs (FELP). Because FELP 
projects have a great deal in common with other family literacy projects, 
the information should be helpful to those working to improve assessment, 
evaluation, and instruction in projects that serve language-minority fami- 
lies. These projects might be funded through a variety of sources. 

This chapter will first present the distinctions between standardized and 
alternative assessments. Next, the special evaluation needs of family 
literacy projects will be highlighted. Finally, the chapter will provide 
overviews of Chapters 11- VI of the handbook. 

* Sec the Glossary on pages 1 37- 1 38 of this publication for definitions of italicized key terms that 
appear in this chapter 
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Standardized and Alternative Assessments 



Alternative approaches 
allow staff members to identify 
what is important in their 
project and select assessment 
strategies that are tailored to 
the unique characteristics of 
the learners and the project. 




This handbook is baseJ on the assumption that successful 
assessment and evaluation depend on using a variety of approaches. 
Some approaches are standardized, such as the Basic English Skills 
Test (BEST) (Center for Applied Linguistics, 1982) or the Comprehen- 
sive Adult Student Assessment System (CASAS, 1982). In contrast, 
alternative approaches refer to procedures and instruments that are 
tailored specifically to the learners, curriculum, and overall design of 
the project. In this section, standardized and alternative approaches 
are described. 

Standardized Approaches 

Standardized approaches rely on instruments that arc created to 
allow for eompari sons between individuals* current achievement and 
the average performance (norms) of selected participants (norming 
group). These approaches offer many advantages to literacy project 
staff; the tests can be obtained readily, administered easily, and scored 
immediately and accurately in a cost-effective manner. Staff can use 
standardized instruments to compare the performance of project 
participants with the norms established by the test developers. 

Standardized approaches, however, provide only part of the 
information needed to document learners* progress. For example, the 
results of standardized measures may not be useful or meaningful to 
staff members if the content of the tests is not related to the goals and 
curriculum of the project. Also, the tests may measure discrete skills 
such as word recognition, but the project may be designed to 
strengthen shared literacy activities between parents and children. In 
addition, standardized tests may be based on a norming group that 
differs from the participants in the project, thereby making it difficult 
to compare the participants' performance with the standard (norm 
group). 

Because standardized measures usually focus on products (i.e., 
responses to specific test items), they may fail to provide useful 
information about importani processes such as enjoying books or 
obtaining needed social services. Furthermore, many teachers may not 
fully understand how to interpret and use the scores. Even when staff 
are fully trainee in the use of standardized tests, the scores may not 
provide enough information about student learning or information that 
is useful for making decisions about the next steps that should be 
taken for designing activities, selecting instructicnal materials, or 
refining projeet goals. 

Alternative Approaches 

Alternative approaches to standardized assessment and evalua- 
tion may be characterized as flexible, representative of the curriculum, 
meaningful to learners, and indicative of learners' acquired ability or 
knowledge (Navarrete, Wilde, Nelson, Martinez, & Hargetu 1990). 
Alternative approaches allow staff to identify what is important in 
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Surveys are used to 
obtain general information from 
large numbers of individuate. 



Interviews are designed 
to collect detailed information. 



Observation measures 
are used for collecting and 
recording infoymation about 
various aspects of the program. 



Performance samples are 
examples of the learners' work 
in selected tasks. 



their project and select assessment strategies that are tailored to the 
unique characteristics of the learners and the project. Unlike standard- 
ized tests, alternative approaches can identify the learners' strengths 
and needs without comparing results with an external standard or 
norming group. Alternative approaches provide multidimensional, 
highly current views of the learners' progress in many different 
contexts. Because the contents of the instruments can be linked 
directly to the curriculum of the project, alternative approaches give 
staff members immediate access to feedback for planning subsequent 
learning activities. 

Four alternative approaches are discussed in this volume: sur- 
veys, interviews, observation measures, and performance samples. 
Other examples of assessment alternatives include portfolios, journals, 
and investigations ( Wrigley , 1 992). To provide a consistent analysis in 
all the chapters, the authors have adopted definitions of the approaches 
that may differ somewhat from the way in which they are discussed 
elsewhere in the assessment literature. The following is a description 
of the four alternative approaches presented in this handbook. 

Surveys are used to obtain general information from large 
numbers of individuals. Questions used in surveys may be open-ended, 
allowing subjects to provide their own responses, or closed-ended, 
requiring subjects to select only from the response choices provided. 
The answers may be recorded by the respondents or by the person(s) 
administering the survey. Examples of surveys are presented in 
Chapter III. 

Interviews arc designed to collect detailed information. The 
interviewer asks questions orally and may make follow-up inquiries to 
clarify or amplify responses. The interviewer may document the 
results by taking notes or tape-recording responses (Bell, 1988). 
Although interviews are usually administered individually, they also 
may be conducted with groups of people. Chapters III and V describe 
how interviews can be used with focus groups, 

Observation a >easures are used for collecting and recording 
information about aspects of the project such as learner characteristics, 
group irteractions, or literacy performance. They may be used for 
closed-ended assessments, such as indicating the learner's level of 
listening comprehension, or they may be open-ended, such as when a 
staff member judges how confident a student appears when participat- 
ing in a cooperative learning group. Chapter IV provides examples of 
observation measures. 

Performance samples are examples of the learners* work in 
selected tasks. For instance, learners may be asked to write a story, 
give an oral pre mentation, participate in a role play, or read a poem. 
Staff members then determine how the task will be documented and 
how the learner's ability to carry out the task will be assessed. For 
example, stories could be collected in written form, role plays could be 
videotaped, and poem readings could be audiotaped. Performance 
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The authors present 
examples of four alternative 
approaches with the 
understanding that project 
staff members will modify the 
examples or create their own 
instruments and approaches 
based on the unique 
characteristics of the learners, 
staff, and program design. 



Any assessment 
approach has limitations, but 
by using a variety of 
instruments, staff can help 
ensure that they will obtain a 
comprehensive view of their 
project. 



samples differ from observation measures in that observation measures 
record only the assessment of learners* behavior or performance. 
Performance samples include the learner's actual work and the results 
of the assessment of the work (for example, holistic scores on writing 
samples). Examples of performance samples can be found in 
Chapter IV. 

A variety of techniques may be used to record the information that 
is collected in any of the four alternative assessments. For example, a 
check list might be used to indicate responses made to questions in a 
survey. A frequency count could be taken of the answers obtained in an 
interview. A rating scale might be used to record judgments in an 
observatic i measure- 
Portfolios ^ould be used to collect and organize samples of 
learners* writii? results of surveys and individual interviews, informa- 
tion collected through observations, and other examples of learners* 
accomplishments. What is included in the portfolio depends on the 
objectives of the project. Staff can make the contents highly personal- 
ized and meaningful by asking the learner to help decide what is placed 
in the portfolio including measures of self-reflection. 

The authors present examples of four alternative approaches with 
the understanding that project staff members will modify the examples 
or create their own instruments and approaches based on the unique 
characteristics of the learners, staff, and program dersign. Project staff 
should consider many factors when selecting an alternative approach. 
For example, open-ended responses may give flexibility to respondents 
but take additional time and energy for staff to analyze. Observation 
measures are useful for documenting staffs judgments but may yield 
different data depending on which staff member is doing the observing. 
Any assessment approach has limitations, but by using a variety of 
instruments, staff can help ensure that they will obtain a comprehensive 
view of their project. They may find it helpful to seek the assistance of 
an experienced e valuator to design and implement alternative assess- 
ment approaches. An evaluator can help staff create instruments that 
are psychometrically sound, analyze the results of the assessments, and 
make recommendations for improving the project. 

This handbook focuses on using alternative approaches to 
(1 ) identify the learners' needs; (2) determine the learners' ongoing 
progress in meeting project objectives; (3) ascertain the overall success 
of the project; (4) give staff immediate and relevant feedback; (5) 
enable learners to assess their own development; and (6) help meet the 
evaluation guidelines established by funding sources of family literacy 
projects. The authors assume that staff members will implement 
alternative approaches in conjunction with standardized measures. By 
creating evaluation designs that integrate alternative and standardized 
approaches, staff will have in-depth information to make decisions 
about project improvement. 
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Role of Evaluation 



An effective evaluation design provides learners and staff 
members with accurate and useful information for designing, 
modifying, and improving their project. In order to create 
effective assessment and evaluation procedures, staff and 
learners need to ( 1 ) clarify the goals and objectives of the 
project; (2) develop indicators of progress in attaining the goals 
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Objectives should be 
clear statements representing 
a range of possible outcomes 
that are refined as the project 
is implemented. 



This volume contains suggestions on how objec- 
tives can be used to help project staff define the 
outcomes or changes that are expected as a result of 
the instructional program. The content of objectives is 
determined by what the learners need, what the staff 
members are prepared to teach, and what is possible to 
achieve within the time and budget constraints of the 
project. 

During the Initial planning of a project, staff 
members can use the development of objectives as a 
way to focus the direction of instruction based on the 
results of the needs assessment. For example, will the 
project concentrate on listening, speaking, reading, or 
writing skills— or some combination? To what extent 
will the learners 1 native language abilities be devel- 
oped? How will the project help parents support their 
children's school achievement? 

Some objectives may deal with clearly observable 
outcomes such as the ability to read or write a piece of 
text; others may specify changes that are more difficult 
to see such as self-confidence, group participation, or 
literacy awareness. Each objective should specify only 
one outcome. Whatever the outcome, an objective 
. should include assessment procedures and instru- 
ments that can be used to measure the extent to which 
the objective is being achieved. 

Objectives should not be immutable, rigid direc- 
tives from the staff to the learners. Nor should they be 
developed by staff without involving the learners or 
assessing their needs. Rather, objectives should be 
clear statements representing a range of possible 
outcomes that are refined as the project is imple- 
mented. Among the myriad literacy-related activities 
that a project might address, objectives specify where 
the staff and learners could begin their collaborative 
effort toward building a successful project. 
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and objectives; and (3) identify the information they need to collect to 
determine the degree to which success has been achieved. 

The focus of the evaluation should be based on the goals and 
objectives that learners and staff members have established. For 
example, if the learners are composing stories they will read to their 
children, the evaluation design should include the procedures for 
collecting information about the learners* compositions and the 
strategies they used for sharing them with their children. 

Assessment versus Evaluation 

In this volume, assessment refers to the use of instruments and 
procedures to gather data on a regular basis. Assessment may focus on 
identifying learners* needs, documenting learners' progress toward 
meeting their own goals, and ascertaining the extent to which the 
project objectives are being met- 

Evaluation, on the other hand, refers to the integration and 
analysis of assessment data at a given point in time for such purposes 
as (1) interpreting learners* needs; (2) developing goals and objec- 
tives; (3) designing the content of the curriculum; (4) selecting 
instructional approaches; (5) monitoring the implementation of the 
project; (6) identifying obstacles to achieving objectives; and 
(7) determining the overall success of the project. An example of an 
evaluation activity is the annual evaluation report that is required of 
projects funded by the Office of Bilingual Education and Minority 
Languages Affairs (OBEMLA), U.S. Department of Education. 

This handbook presents evaluation as pari of a comprehensive 
process of planning, implementing, and improving a project (see 
Chapter II for a description of the planning process). Evaluation 
should not be an isolated event that is conducted at the end of each 
year. Rather, evaluation is an ongoing, collaborative effort by the 
learners, staff members, and evaluator to clarify learners* needs, 
refine project goals, design curriculum content, and develop instruc- 
tional methodology. The essence of the evaluation is the participants' 
specification of what constitutes success in their project, what 
approaches they will use to assess the level of success they have 
achieved, and how evaluation results will be used to improve the 
project. 

Diversity in Family Literacy Projects 

Family literacy projects address the needs of adults and children 
who are strikingly diverse in terms of their linguistic and cultural 
backgrounds, the characteristics of their families, the demography of 
their communities, and their reasons for wanting to improve their 
literacy abilities. This section provides some examples of the diversity 
in families and in the design of literacy projects (see also Holt & Holt, 
in press). 
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Participants, Location, and Schedule 

Family literacy projects are characterized by diversity in terms of 
participants, location, and schedule. Many different groups are served 
such as American Indian, Cambodian, Chinese, Haitian, Hispanic, 
Hmong, Korean, Russian, Samoan, and Vietnamese. The ages of 
participants also vary am ng projects. Some projects focus on young 
adults who are not in school. Others enroll only adults, some of whom 
may be parents. Still other projects include opportunities for parents 
and children to learn together in intergenerational activities. Projects 
may be located in large cities or rural areas, operated during the day or 
evening, or conducted in schools, community centers, or the partici- 
pants 1 homes. 

Program Design 

Literacy projects also differ in the way in which they have 
responded to the program requirements of their sources of funding. 
Most projects include activities that promote the learners' acquisition 
of English. Some projects may focus on language development, 
whereas others integrate language and life skills. Other projects focus 
on oral communication as a prelude to literacy, and others explicitly 
address the development of reading and writing. Some projects include 
activities to help learners develop their abilities in English and their 
native languages. Other projects assist participants to improve their 
knowledge of topics such as mathematics, nutrition, the U.S. educa- 
tional system, and social services. Subject matter may be taught in 
English or a combination of English and the learners* native language. 

Home-School Linkage 

Projects are further distinguished by the degree to which they 
include instruction on how parents and family members can facilitate 
children's success in school. Some projects attempt to create a close 
link between the parents* learning activities and the educational 
experiences of their children. For example* a project might be designed 
to help parents learn skills that would enable them to serve as volun- 
teers at school or to work with their children at home in school -related 
activities. 

Characteristics of Family Literacy Projects 

The special characteristics of family literacy projects necessitate 
the use of assessment and evaluation approaches that are specifically 
tailored to the needs of learners and staff. Alternative approaches yield 
information that learners and staff can use to set project goals, decide 
on instructional priorities, document successes and failures, and 
improve future activities. In December 1990, staff members from 
FELP projects in California held a coordination meeting in which they 
analyzed the unique aspects of family literacy projects and considered 
innovative approaches to evaluation that would meet their needs. This 
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section is based in part on the discussions at the seminar. (See Pro- 
ceedings from the Family English Literacy Seminar, 199 1 , for a 
summary of the results of the seminar.) 

Enrollment and Attendance 

Unlike students in elementary and secondary schools, learners 
in most family literacy projects enroll voluntarily. However, before 
they can enroll and benefit from services, they must know that a 
project exists. Because participants may enroll in a project at any 
time, staff members must be prepared to collect baseline data when- 
ever new participants enroll. After enrollment, many adult learners 
may have difficulty attending project activities because of scheduling 
conflicts caused by personal, family, or job-related issues. 

Assessment and evaluation strategies need to take into account 
the uneven enrollment and attendance patterns that characterize many 
projects. Ongoing use of alternative approaches is more effective 
than infrequent pre- and posttneasures for enabling staff to identify 
obstacles to the learners' participation and make adjustments that will 
facilitate enrollment and attendance. 

Multiple Contexts, Goals, and Needs 

Adult learners face demands in the home, community, school, 
and work place. Depending on the requirements of these contexts, 
learners come to the literacy project with different goals and needs 
for acquiring language and literacy. Some adults may want to learn 
English to work in a restaurant. Others may be interested in opening a 
business. Some may need to learn English for a job interview, and 
others may want to help their children write folk tales in their native 
language. Still others may have more general goals such as learning 
English to co.nmunicate and negotiate with native speakers. 

Adult learners also have different levels of knowledge about the 
school, community, and work place. Staff members need assessment 
instruments to obtain accurate information about the learners' 
knowledge and needs in order to plan appropriate instruction. Alter- 
native approaches, which can be tailored specifically to the learners' 
language and cultural backgrounds, provide staff with highly special- 
ized and useful information. Staff members can use the assessment 
results for designing objectives and selecting instructional content 
and methodology that will validate the learners' prior experiences 
and facilitate their success in the project. 

Literacy and Biliteracy 

Most participants in family English literacy projects have a 
native language other than English. However, families differ accord- 
ing to the purposes for which they use their native language and 
English and the degree to which they depend on each language to 
carry out daily tasks. For example, in some families, spouses may use 
their native language with each other but use a mixture of English and 
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theiv native language with their children. Learners also have different 
levels of proficiency and educational experiences in their native 
language. Some learners may have received a high school diploma, 
but others may have come from countries in which no formal school- 
ing was available. 

According to Quintero and Huerta-Macfas (1990), to provide 
appropriate instruction, staff need to know the parents* and children's 
proficiency levels and uses of their native language and English. The 
lack of standardized instruments to assess proficiency in languages 
other than English makes the use of alternative approaches essential 
for determining the participants' needs, competencies, and growth in 
their native languages. Chapters 111 and IV contain suggestions for 
assessing native language proficiency. 

Parent and Child Outcomes 

When family literacy projects create objectives and instructional 
activities for parents and children, staff members need to monitor 
changes based on the special characteristics of two very different 
kinds of participants. If parents and children are learning together in 
intergenerational activities, the evaluation design needs to examine 
the effect that parents and children have on each other's success in 
language learning, literacy development, academic achievement, and 
other outcomes identified by the projects. Alternative approaches to 
assessment and evaluation allow staff to collect information that can 
provide answers to important questions about the progress that parents 
and children are making in teaching and learning from each other. 

Staff Knowledge 

Staff need to know about the cultural and linguistic backgrounds 
of adult learners to design and implement effective strategies for 
recruiting participants, identifying learners' needs, and designing 
appropriate instructional activities. Staff members may need training 
in second language acquisition, bilingual development, and adult 
education so they can create effective activities for language-minority 
adult learners. 

Because alternative approaches to assessment and evaluation are 
developed in close consultation with staff and learners, they are 
effective for helping staff improve their knowledge of learners and 
assessing the ongoing progress of the project. Chapter II emphasizes 
the importance of the development of a literacy framework, that is, 
staff members' and learners' shared understanding of the learners' 
literacy needs, program goals, curriculum and methodology, and 
approaches for assessing and evaluating literacy-related activities. 

Issues En Evaluating Family Literacy Projects 

Projects that are funded under ESEA Title VII are guided by the 
evaluation requirements set forth in appropriate federal statutes and 
regulations. Other funding sources have their own evaluation require- 
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merits. Slui'f members need to determine how they can construct an 
evaluation design that meets the requirements of their funding source 
and provides useful information for assessing learners and evaluating 
the project. By combining alternative and standardized measures, staff 
members and learners will improve their access to useful and accurate 
infemation for making decisions about their project. 

In general, the requirements of most funding sources specify that 
three areas be considered in designing the evaluation: ( I ) student 
outcome data; (2) technicaJ standards; and (3) implementation data. 
Outcome data refer to what the students learn during the project. The 
students in FELP projects may be adults, parents, parents and their 
children, or out-of-school youth. Project staff need to examine the 
regulations and determine how the required data can be obtained for 
all the participants in the project. Chapters II through V of this volume 
emphasize the importance of collecting and analyzing outcome data 
based on the special features of the project. Issues regarding the 
evaluation of implementation data that are unrelated to student 
outcomes (for example, staff development) are not addressed in this 
volume. 

This section presents some of the issues related to evaluating 
family literacy projects. Although the issues are based on the experi- 
ences of staff members' evaluation of ESEA Title Vll FELP projects, 
the information should be helpful for individuals working in any 
family literacy project. 

Comparison Group 

At least part of the evaluation design of a literacy project may 
include the assessment of the educational progress of project partici- 
pants against an appropriate nonproject comparison group. Staff 
members can meet this requirement with a standardised test by 
comparing the participants' progress with the test's norming group. 
To complement the results of the standardized test, staff could use 
alternative approaches to assess the learners' progress without having 
to compare their performance with a nonproject comparison group. 

Instead of standardized tests, or in addition to them, staff may 
use alternative assessment approaches with a nonprojeel comparison 
group. For example, the current participants' performance can be 
compared with ( 1) the performance of individuals who are not partici- 
pating in the project and have similar characteristics to the partici- 
pants or (2) baseline data that had been collected at an earlier point in 
the project on the current participants and all other participants who 
were enrolled at an earlier time. Staff should note they may not need 
to compare the results of every assessment in the project with a 
nonproject comparison group. Chapter V presents suggestions for 
using alternative approaches with a nonproject comparison group. 

Representativeness of the Findings 

Evaluation findings should apply to the participants, schools, or 
agencies served by the project, In other words, the conclusions made 
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in an evaluation report should be derived from data on learners served 
by the project and on the full range of services provided by the project. 
Although absenteeism and transiency may prevent staff from obtaining 
assessment data on all learners, data should be collected from a sample 
of learners that is representative of the learners served by the project. 
Similarly, staff should obtain infonnation about the learners* perfor- 
mance in the major components of the project such as language devel- 
opment, literacy, parenting, and cultural adaptation. Alternative 
approaches enable staff to tailor assessment and evaluation to the 
characteristics of the learners and the services of the project, thereby 
helping staff to ensure that the evaluation findings are indicative of 
what is actually happening in the project. 

Validity and Reliability 

Alternative instruments and procedures should be valid; that is, 
ihey should measure what they claim to measure. Assessments also 
should be reliable; in other words, they should produce similar results 
consistently. Staff members need to devote time and resources to 
develop and field test alternative approaches in order to document their 
validity and reliability (see Guidelines for Developing Reliable and 
Valid Alternative Assessments, EAC-West, 1991). Chapter V ad- 
dresses issues of validity and reliability in the evaluation design. 

Multiple Measures 

A variety of alternative approaches should be used for conducting 
intake and initial assessments, monitoring progress, and assessing the 
overall success of the project. Multiple instruments and procedures can 
provide staff with a comprehensive view of the learners, the progress 
they are making, and the effectiveness of the activities of the project. 

Overview of the Handbook 

This handbook is intended to assist staff members in family 
literacy projects to design and implement alternative approaches to 
assessing and evaluating their project. Each chapter addresses a 
different aspect of assessment and evaluation. 

Chapter II contains a model for integrating program planning, 
implementation, and evaluation activities, A central element of a 
successful project is a conceptual framework for literacy thai is shared 
among the staff and learners. The framework represents the staff's 
collective thinking regarding the definition of literacy, appropriate 
instructional approaches, the respective roles of staff and learners, and 
other factors that influence the design, implementation, and evaluation 
of the project. The author t X)ints out how the program planning and 
evaluation processes operave in synchrony when all participants share a 
common understanding of the design and focus of the project. 

Chapter 111 addresses initial assessment, including needs assess- 
ment, intake, and placement. The chapter contains examples of alterna- 
tive approaches that have been used successfully for initial assessment: 
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surveys, interviews, and writing samples. The author suggests strate- 
gies lor using initial assessments to yield accurate baseline informa- 
tion about the learners so staff can design appropriate learning activi- 
ties and determine the extent to which the learners are making 
progress in the project. 

Chapter IV describes how alternative assessment and evaluation 
approaches can be used to document learners* progress toward 
meeting instructional objectives. Three alternative approaches are 
presented in detail: observation measures, interviews, and writing 
samples. The author emphasizes the importance of ongoing assess- 
ment for ( 1) helping staff members determine the extent to which their 
instructional efforts are leading to positive results and (2) showing 
learners that they are making progress toward meeting their own 
goals. 

Chapter V provides suggestions for using surveys, performance 
samples, focus groups, and observation measures to demonstrate 
progress toward achieving the goals and objectives of the project. The 
author presents strategies for collecting, analyzing, and reporting data 
obtained with each of the alternative approaches. The chapter under- 
scores the importance of staff members' and learners' use of the 
evaluation data for ( ' ) making decisions that will refine the goals, 
objectives, curriculum, and overall design of the project and (2) 
determining the progress made toward attaining the goals of the 
project. 

Chapter VI describes the process that was used in designing and 
field testing the handbook. The author uses the planning model 
presented in Chapter II to illustrate how the contents were developed 
and how feedback was collected during the one-year field test. The 
chapter also contains information that readers requested about stan- 
dardized tests and work place literacy. 

Summary 

This volume is designed to help staff members use alternative 
approaches to assessing and evaluating their literacy project. No 
single approach to assessment can provide a comprehensive view of 
what is happening in a literacy project. By integrating alternative 
assessment approaches with standardized assessment measures, staff 
can obtain accurate and complete information for evaluating their 
project. Four alternative approaches are discussed in this volume: 
surveys, interviews, observation measures, and performance samples. 
Each chapter presents strategies for designing and implementing 
alternative approaches that are tailored to the needs of individual 
projects. The chapters follow a sequence that might be used by staff in 
addressing assessment and evaluation issues: (1) planning the project 
and creating the evaluation design (Chapter II); (2) determining 
learners' needs and establishing baseline data (Chapter III); (3) 
assessing learners* progress toward meeting project objectives 
(Chapter IV); (4) collecting, analyzing, using, and reporting data 
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obtained with alternative approaches (Chapter V); and (5) understand- 
ing the role of planning and evaluation in creating and improving a 
project (Chapter VI), 
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CHAPTER II 

INTEGRATING 
PROGRAM PLANNING, 
IMPLEMENTATION, 
AND EVALUATION 



"If you want something done right, do it yourself best expresses the theme of 
this chapter. Much has been written regarding the shortcomings and failurts of 
traditional assessment* and evaluation to meet the needs of adult literacy 
projects. Current research and practice in adult literacy suggest ways of 
improving the utility and relevanee of assessment and research efforts. 
However, the most effective alternatives to traditional approaches are created 
by staff members and adult learners as they collaborate to improve their own 
literacy projects. 

The purpose of this chapter is to describe a process for planning, implement- 
ing, and evaluating literacy programs for limited-English- proficient (LEP) 
adults, parents, and children. The process emphasizes the importance of using 
alternative approaches for designing and evaluating literacy programs. 

The chapter is divided into four sections: ( 1 ) an explanation of how assess- 
ment and evaluation procedures are determined in part by how learning is 
defined; (2) a discussion of the requirements for an alternative assessment and 
evaluation design; (3) a deseription of the importance of a well defined 
literacy framework for the project; and (4) a presentation of a planning 
process that project staff can use as a guide to develop the instructional 
program and an appropriate assessment and evaluation design. 

• See Ihe Glossary on pojes 1 37-1. IK nf this publication for definition* of iialh-izrd key terms that 
appear tn (htt chapter. 
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How learning is defined helps determine how literacy is defined 
and evaluated. The way staff members define learning influences 
what assessment information they collect, how they collect it, and 
how tiicy use it in the project. Currently there are two views of 
learning with decidedly different implications for data collection and 
use. 'The first is a traditional view of learning that has dominated 
education since the turn of the century. The second view, bolstered by 
research in cognitive psychology, is helping to redefine evaluation. 

Traditional View of Learning 

Wolf, Bixby, Glenn, and Gardner (199 1) explain how the field of 
evaluation has come to rely almost exclusively on standardized tesis. 
Since the turn of the century, intelligence has been viewed as a fixed 
attribute (unchangeable, like skin color), able to be ranked (some- 
where on a normal curve), and genetically determined. Educators who 
hold this view of intelligence classify students into specific groups 
based on the assumption that they have different abilities and there- 
fore need a special curriculum. Such approaches can lead educators to 
assign students permanently to groups according to their perceived 
abilities. Because intelligence is seen as a fixed attribute, learning is 
viewed as one dimensional, with individuals progressing through a 
series of discrete steps from simple entry level skills to full mastery. 
The perception of learning as molecular and hierarchical has guided 
the development and use of standardized tests. 

According to Wolf and others, "Standardized tests ... are 
constructed so that only a few can score high; they have yielded 
artifacts rather than authentic samples, promoted an isolated view of 
learning, and treated assessment as a matter of pure measurement" 
(Wolf, et al.» 1 991 , p. 47). The tests contribute to the view that assess- 
ment results are end products and that cognition is an individual, 
isolated activity. Test developers have often created instruments that 
are more effective at ranking students than identifying where they are 
developmentally. The ranking of students results in various negative 
consequences, not the least of which is that those who rank at or near 
the bottom are ignored in the learning process or otherwise stigma- 
tized. Contemporary research in developmental and cognitive psy- 
chology, however, has questioned traditional assumptions about 
learning and suggests decidedly different directions for data 
collection. 

Learning Redefined 

Current research suggests that intelligence is not a fixed phe- 
nomenon but a dynamic, global capacity. All human activity requires 
that "the learner actively develop knowledge by observing, inferring, 
generating rules, and building theories" (Wolf, et al., 1991, p. 48). 
Learning should be viewed not as the development of a series of 
discrete skills, but as a process involving "qualitative and uneven 
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shifts in understanding" (Wolf, et al., 1991, p. 50). Learning 
represents a series of ever increasing, complex uudersiandiugs 
as an individual observes, theorizes, tests, observes further, and 
readjusts the theory. 

Learning is not only the acquisition of discrete informa- 
tion; it is also the organization of information as a basis for 
further learning. Furthermore, learning does not occur in 
isolation. It is affected profoundly by external stimulation from 
other individuals, environmental pressures, or resources. This 
redefinition of learning suggests that staff members in family 
literacy projects need to consider carefully the kind of evalua- 
tion data that is collected and the methods of gathering data. 
Assessment procedures should be used to determine the 
learner's ability to construct and apply knowledge for a variety 
of purposes. 

Importance of a Literacy Framework 

Assessment, evaluation, and other components of a 
literacy project need to work together in a mutually supportive 
manner. One way to coordinate the components is to create a 
literacy framework. A project's literacy framework may be 
described as the staff members* and learners* shared under- 
standing of learners* literacy needs, program goals, curriculum 
and methodology, and approaches for assessing and evaluating 
literacy development 

To design and implement alternative assessment instru- 
ments, staff need to identify outcomes and changes that consti- 
tute the learners* success in the project. In other words, how 
will the staff and learners know that progress is being made in 
attaining the objectives of the project? The staff members' and 
learners' shared understanding of the nature of literacy and the 
specific outcomes that the participants are working toward 
represent key parts of the project's literacy framework. 

As was pointed out in Chapter I, family literacy projects 
have different program designs. Some projects may help 
parents improve their individual language abilities, whereas 
others focus on shared literacy activities between parents and 
their children. Developing a literacy framework is One of the 
first steps the staff and adult learners can take to Create a 
common understanding of the role of literacy in the project and 
in the learners' lives. 

A Collaborative Approach to 
Planning and Evaluation 

One of the keys to linking the project's literacy frame- 
work, program design, and assessment and evaluation proce- 
dures is a collaborative approach to planning and evaluation. 
This section presents a rationale and process for creating a 
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literacy framework for the project and describes the steps involved in 
a collaborative approach to planning and evaluation using examples 
taken from actual FELP projects in California. 

Contexts for Literacy 

Individuals and language groups differ according to the ways in 
which they use language and literacy and the settings or contexts in 
which they read and write (Lytic, Belzer, Schultz, & Vannozzi, 1989). 
Thus, adult learners have different goals and needs for literacy devel- 
opment. Learners may need to be able to write notes to their children's 
teacher, interpret monthly bills and bus schedules, decipher product 
labels, help their children with school work, read a repair manual, or 
complete an application for admission to a training program, Alterna- 
tive assessments offer staff the advantage of being able to measure the 
learners' ability to use literacy in contexts that are relevant to their 
daily lives. 

Literacy is best viewed as a range of practices or activities 
(Scribner, 1987). In a recent research study conducted by Wrigley and 
Guth (1992), ESL literacy instruction was defined as "supporting 
language-minority adults with little English and little formal educa- 
tion in their efforts to understand and use English in its many forms 
(oral and written, including prose, document, and quantitative lit- 
eracy), in a variety of contexts (family, community, school, work), so 
that they can reach their fullest potential and achieve their own goals 
(personal, professional, academic)" (p.7). Chapter IV provides 
detailed suggestions for how curriculum and assessment can be 
matched with this holistic view of literacy. 

Project staff need to work closely with adult learners to design 
assessment procedures that will identify the types of reading and 
writing the learners are doing and their individual "purposes and 
aspirations" (Lytle, et al., 1989, p. 55). The procedures should help 
staff and learners examine their own views of learning, literacy, and 
how literacy is used in their daily lives. Staff and learners also can 
collaborate in designing and implementing evaluation activities 
including analyzing and interpreting evaluation data. Auerbach (1992) 
emphasizes that adult learners bring to the project their own goals, 
prior literacy experiences, and attitudes toward reading, writing, and 
learning, all of which interact and affect what and how they will learn. 
Such background information needs to be used to create meaningful 
assessment and evaluation activities in the project. 

Lytic ct al. ( 1989) designed a framework to guide the develop- 
ment of assessments and evaluations that consider the diverse needs 
and contexts of adult learners. Their framework has four dimensions: 
I ) literacy practices; 2) reading, writing, learning strategies, and 
interests; 3) perceptions of reading and writing, leaching and learning; 
and 4) goals. Literacy practices or activities refer to how, when, and 
why learners use literacy in their lives. After the staff and learners 
have identified Ihe role of literacy in various contexts, assessment and 
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evaluation approaches can be used to measure changes in the way 
learners use literacy as the project is implemented. 

The framework outlined by Lytle et al. (1989) may be helpful to 
FELP participants who are designing their own literacy framework. 
For example, the literacy framework should include a definition of 
literacy; an identification of the role that literacy plays in learners* 
lives in various contexts; documentation of the learners* goals for 
literacy development; the focus of curriculum content and activities; 
and the basic direction of the assessment and evaluation procedures 
Collaboration among staff and adult learners in developing a literacy 
framework enables all participants in the project to inerease their 
understanding of literacy and improve their ability to use literacy to 
achieve their own goals. 

The next section presents suggestions for designing a literacy 
framework that is based on the learners* needs. The literacy frame- 
work and planning process eontain elements that were borrowed from 
the work of Lytle, Auerbach, and others. The information is presented 
to encourage projects to begin the process of creating learner-cen- 
tered, participatory approaches to literacy development. The reader 
should understand, however, that this chapter and others in the volume 
contain suggestions, such as developing project objectives and using 
ready-made curriculum, that some advocates of participatory literacy 
approaches might consider inconsistent with authentic learner- 
centered programs. 

Developing A Literacy Framework Through Collaborative 
Planning 

The development of a literacy framework is the first step in 
designing alternative assessment and evaluation procedures. The 
literacy framework is comparable to model curriculum standards that 
educators can use as a foundation for developing programs that meet 
local needs (see California Department of Education, 1 992). 

One of the assumptions of an effective literacy framework is that 
adult learners acquire abilities and knowledge in a spiral versus 
sequential fashion, with increasing levels of proficiency and under- 
standing emerging over time. As staff and learners work together to 
create the framework and related curriculum, they also can develop 
project-based assessment and evaluation procedures to document 
progress toward meeting objectives and making improvements in the 
program. The importance of the literacy framework as a foundation 
for assessment is underscored by Lytle, Marmor. and Penncr: "Our 
starting point was the understanding that assessment must correspond 
to program philosophy and goals, since assessment procedures 
embody and thus convey particular concepts about literacy" (1986, 
p. 22). 

At a meeting of selected California FELP projects, staff mem- 
bers attempted to reach consensus on a definition of literacy. The 
results of their discussion are summarized in the seven points listed 
below (Results of the Focus Group Meeting* 1991, p. 4). 
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1 . Literacy is Finding meaning from print in order to fulfill the learners* 
academic, psychological, political, and social goals. 

2. Literacy includes a range of activities or practices in multiple contexts 
with a variety of people. It may involve the use of more than one 
language. 

3. Contexts differ with respect to the complexity of literacy demands. 

4. Literacy activities include reading, writing, and learning strategies. 

5. Literacy in more than one language strengthens learners* linguistic and 
cognitive development and increases their sociocultural knowledge. 

6. The development of literacy is influenced by the language, culture, 
status, experiences, and perceptions of learners and teachers. 

7. The concept of literacy is based on the teachers* and learners' values 
and beliefs. 

These seven statements reflect the staff members' underlying 
assumptions about literacy. From this initial framework, several 
implications can be drawn. First, literacy should be developed in ways 
that are meaningful to learners. Second, the literacy curriculum needs 
to address the ways in which learners want to use literacy. Third, 
learners should be deeply involved in the development of the curricu- 
lum content. Fourth, literacy instruction is a collaborative effort 
between the learner and the teacher. 

In an actual project, the literacy framework would be refined as 
the project is implemented and evaluated. Initial aspects of the frame- 
work might be quite general; however, as the staff and learners 
continue to work together, they will improve their understanding of 
the nature of literacy and how it can be fostered in the project. 

To support a literacy framework such as the one described 
above, assessment and evaluation procedures should do the following: 

1. Focus on the learners — that is, accurately identify learners* 
strengths and needs and assess learners* progress in the project; 

2. Document how improvement in learners' literacy development 
affects their children, work responsibilities, and community 
activities; 

3. Rely on a variety of instruments and strategies for collecting data; 

4. Reflect changes in learners* perceptions, attitudes, and behavior 
inside and outside the classroom; 

5. Collect and analyze data on a continual basis throughout the 
project. 

Process and Product Evaluation 

Alternative assessment and evaluation procedures raise a num- 
ber of questions related to process and product evaluation. A process 
evaluation focuses on how a project is implemented. A question that 
might be addressed in a process evaluation would be the following: 
How and to what extent are learners involved in identifying their 
literacy goals and objectives, designing the learning activities of the 
project, and assessing their progress in attaining their goals? 

A product evaluation is concerned with the effect of the project 
on the adult learners. A product evaluation addresses such questions 
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as the following: To what extent have adult learners increased their 
language proficiency? To what degree have they attained their per- 
sonal literacy goals? To what extent does increased literacy develop- 
ment affect how learners (1 ) perceive themselves; (2) relate to their 
spouses or other family members; (3) respond to friends and co- 
workers; and/or (4) use their literacy skills in new settings or for 
different purposes? 

To answer these and other questions, learners need to be assessed 
in different ways. The alternative approaches presented in Chapters 
HI, IV, and V include surveys, interviews, observation measures, and 
performance samples. Surveys and interviews of family members, 
friends, co-workers, employers, and classmates can provide valuable 
information regarding learners' needs, goals, and preferences for 
curriculum content. Observation measures can be used by staff to 
assess the learners' progress in acquiring language and literacy. 
Performance samples of learners' work, such as writing samples, tape 
recordings of oral readings, and journals, are useful for documenting 
learners' language development. By using a variety of alternative 
approaches, staff and learners can develop a comprehensive picture of 
what is happening in the project and satisfy the needs of both process 
and product evaluation. 

A Planning Process Model 

Developing a quality instructional program and evaluation 
design requires a thorough planning process involving the collabora- 
tion of staff members and adult learners. Figure 1 illustrates each of 
the steps of the process. The model is heuristic; that is, it is shown as 
an idealized guide for planning. In actual practice, the tasks are not 
always implemented in sequence. Sometimes steps are skipped and 
then gone back to. At other times, steps are addressed simultaneously. 
Most important, the steps are interdependent; that is, what happens in 
implementing one step influences activities in other steps. 

The planning process is cyclical; the knowledge gained by 
implementing one step is used to inform decisions related to other 
steps. The implementation of the steps should be a collaborative effort 
between staff and adult learners. Figure 1 illustrates that effective 
planning is not a one-time effort, but rather a series of activities thai 
occur throughout the life of the project. By continuously examining 
the needs of the learners, staff, and the community in general, staff 
members can create instructional activities, assessment strategies, and 
evaluation procedures that are highly responsive to the learners and 
the conditions under which the project is being implemented. 

Staff and learners need time to organize and implement the steps 
in the model. Although the entire process cannot be implemented fully 
in any one year, staff and learners should work through as many steps 
as possible, implementing fully when feasible, and laying the ground- 
work for other steps that might be addressed later. 
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The process is designed to give staff and learners a blueprint for 
continuously collaborating to design and refine their project. Visualiz- 
ing the entire process enables staff to schedule time and resources for 
dealing with each step, For example, teachers may devote class time to 
activities in which learners would discuss the relevance of instruc- 
tional content to their personal goals and the extent to which they 
believe they are making progress. Ideally, staff members should be 
able to meet weekly or biweekly lo assess and clarify goals, objec- 
tives, instructional activities, and other issues. 

Staff members will benefit from planning activities that result in 
( I ) clarifying adult learners 1 needs; (2) defining the dimensions of 
literacy addressed by the project; (3) refining curriculum content; and 
(4) selecting appropriate instaictional methods. By establishing a 
consensus about the important aspects of the project, staff members 
can coordinate their responsibilities more successfully. They also can 
provide a clear orientation to new staff members and adult learners 
who join the project in midstream. A literacy framework becomes a 
useful reference for communicating the program's philosophy, goals, 
and procedures to others. Specifying clear goals and objectives helps 
participants collaborate to create a consistent, coherent instructional 
program that responds lo learners' needs and supports their continued 
growth. A collaborative planning process enables participants to 
identify changes that will improve the project based on the collective 
resources of staff and learners. 

Applying the Planning Process Model 

Collaboration is facilitated when individuals establish a trusting, 
supportive relationship with each other. Staff members need to model 
the cross-cultural sensitivity that is essential for developing positive 
relationships. Positive interpersonal relations can be enhanced by 
creating activities designed to build a cooperative spirit among staff 
and adult learners. Team building exercises are useful for developing 
participants' ability to recognize team members' contributions and 
realize the value of helping and receiving help from one another. 
Developing cooperative skills will improve the participants' ability to 
implement the planning process model. (For more on improving 
human relations in the work place, see Johnson, 1987.) Chapter IV 
contains procedures for assessing adult learners' group participation 
skills. 

The following are descriptions of each step of the planning 
process model and examples of how the step has been addressed by 
FELP projects in California. These descriptions represent only some 
of the many approaches that are possible for designing, implementing, 
and evaluating family literacy projects. 

Q Develop the goals of the project. 

Goals are broad, general statements of the intent, purpose, or 
expected outcome of a project. Usually the goals of a literacy project 
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are derived in part from the purposes of the program as delineated by 
the ESEA Title VII statute for FELP. 

Goal statements help staff members design the needs assessment 
activities that take place in Step 2 of the planning process. In the initial 
stages, goals are tentative, because they may need to be adjusted based 
on what the staff ]earn from assessing the needs of the learners. As the 
project is implemented (Steps 4-12), the staff may need to adjust 
further the goals of the project. 

The goals developed by the FELP project at the Santa Clara 
County Office of Education (1989-92) address the major purposes of 
FELP; that is, developing the participants' English language skills and 
the patents' ability to help their children succeed in school. The 
following goals were designed to reflect the unique interests of the 
learners and staff in the Santa Clara area (Proceedings from the Family 
English Literacy Seminar, 1 99 1 , p. 62). 

1 . Assist LEP Hispanic and Southeast Asian parents and family members 
to achieve literacy and competence in the English language; 

2. Provide parents and family members with knowledge of techniques 
designed to assist their children in developing literacy skills in English 
and appreciation of reading; 

3. Involve parents and family members in direct instructional and support 
roles related to their children's skill development and participation in 
English literacy; and 

4. Support parent/community /school communication and the development 
of parents' leadership, advisory, and advocacy roles. 

The goals represent the focus of the project and reflect the 
choices that the project staff have made withm the flexibility of the 
funding source and the needs of the participants as anticipated by the 
staff. 

© Conduct the needs assessment. 

Assessing the needs of prospective learners is a vital step in the 
planning process. During this step, staff obtain valuable information 
about the learners, their families, and their communities. The first 
phase of the needs assessment usually is conducted as part of the 
proposal for funding. The questions asked of learners are based on the 
goals that staff intend to pursue in response to the funding guidelines. 
After receiving the grant, staff members need to conduct more in-depth 
needs assessments, the results of which may cause staff to clarify or 
modify the goals. 

Before the FELP project in the Lincoln Unified School District 
(1990-93) designed its objectives and instructional activities, staff 
administered a survey (alternative assessment instrument) to a repre- 
sentative sample of the Southeast Asian families that the district 
intended to serve. (Chapter 111 contains a detailed explanation of the 
needs assessments that were conducted by staff at Lincoln.) The results 
of the survey indicated that the parents wanted services to be available 
near their apartment complex. Therefore, the staff established a family 
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education center in a vacant apartment unit located in the middle of a 
two-block area of five apanment complexes occupied by Cambodian 
and other Southeast Asian families. 

The staff of the FELP project ai the National Council of La Raza 
( 1988-91 ) used writing samples produced by project participants as 
pan of its needs assessment. The adult learners wrote the sample in 
Spanish or English, depending on their language proficiency. The staff 
used a holistic scoring technique to score the learners* writing. The 
samples provided important information for refining objectives, 
developing instructional activities, and generating baseline data for 
comparing and evaluating changes in the learners* writing ability at a 
later time in the program. Chapter IV contains more information about 
using writing samples for assessment. 

The FELP project in the Whittier Union High School District 
( 1 990-93) used data obtained from the academic records of the aduK 
learners" children to design objectives and instructional activities. 
These data enabled staff to identify high school students who were 
experiencing academic problems and to design project activities that 
would help parents improve their children's academic performance 
Information gathered through needs assessment activities can help 
staff design project objectives (Step 3) and develop curriculum and 
instructional activities (Step 4). 

@ Design instructional objectives. 

Objectives can help staff specify changes or outcomes that 
learners may experience in their language abilities, parenting strate- 
gies, cultural adaptation, or other important areas as a result of the 
instructional program. Some objectives may deal with clearly observ- 
able outcomes such as speaking and writing; others, however, may 
specify changes that are more difficult to see such as self-confidence, 
group participation, or literacy awareness. Each objective should 
include procedures and instruments that can be used to measure the 
extent to which the objectives are being achieved. (Chapter I contains 
additional information about objectives.) 

Initially, objectives are based on information collected during 
the needs assessment. As the planning process unfolds and more 
information is collected from the learners, objectives and related 
assessment instruments may need to be refined. Ultimately, objectives 
are determined by what the learners need, what the staff are prepared 
to teach, and what is possible to achieve within the time and budget 
constraints of the project. 

The following arc examples of objectives that use alternative, 
nonstandardized assessment measures {Family English Literacy 
Program: ESEA Title V 11 Application, 1991, pp. 26-29). 

1 . By the end of each project year, participants will improve their 

receptive (listening and reading) and productive (speaking and writing) 
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skills at leasi one level as assessed by an observation measure and 
writing sample. (This objective contains four subobjectives that 
specify outcomes and measurements for listening, reading, speaking, 
and writing.) 

2. By the end of each project year, participants will increase by at least 
four ihe number of literacy activities they do with their children as 
assessed by individual interviews. 

^ Develop the curriculum and instructional activities. 

Instructional activities are based on the learners' needs and the 
project's goals and objectives. For example, the staff of the FELP 
project in the Parlier Unified School District (1989-92) responded to 
the large numbers of Spanish- speaking families in the district by 
designing activities in which parents use original stories from Latin 
American countries tor story telling activities with their children. 

The FELP staff of the Fremont Unified School District (1988- 
91 ) organized instructional activities for parents that used children's 
stories from the district's adopted list of core literature. By becoming 
aware of the literature that their children studied in school, the parents 
gained confidence in talking with their children about school activi- 
ties. Chapter IV contains more suggestions for developing instruc- 
tional activities and designing ongoing assessment instruments and 
procedures. 

© Identify obstacles, 

"Forewarned is forearmed" accurately describes the value of 
anticipating potential obstacles to planned activities. Knowing about 
possible impediments helps staff and learners anticipate and refine 
future activities by modifying plans and elaborating strategies to 
overcome hurdles. At a coordination meeting of 14 California FELP 
projects in December 1990, participants identified obstacles they 
faced with regard to 1 ) recruitment of participants; 2) maintaining 
enrollment; and 3) staff recruitment and training. Next, they shared 
strategies they had found successful in addressing these problems. 
The I olio wing arc results of their discussions [Proceedings from the 
Family English Literacy Seminar, 1991, pp. 15-20). 

I , Obstacles to recruiting participants: 

a. Schools have incomplete or inaccurate lists of limited-English- 
proficient students. 

b. Printed flyers advertising classes are ineffective because parents 
may not be literate in English or their native language. 

c. Attending school may be a new experience for parents who have 
liltle or no education in their native countries. 

d. Parents are reluctant to visit the school because they are not knowl- 
edgeable about the educational system. 

e. Students who qualify for the Greater Avenues for Independence 
(GAIN ) program may not participate in the project if classes do not 
meet GAIN requirements. 
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f. Some parents face cultural barriers that restrict their participation. For 
example, in some culiure:.. wives cannot attend class without their 
husband's permission. 

g. Child care is not available. 

2. Obstacles to maintaining enrollment: 

a. Participants must move frequently because of seasonal jobs. 

b. Participants' overtime work prevents them from attending classes. 

c. Responsibilities at home prevent regular attendance — for example, 
child care, family problems, and transportation. 

d. Participants may become discouraged if the project activities do not 
meet their needs. 

3. Obstacles to staff recruitment and training: 

a. Siaff who are knowledgeable about adult ESL, parenting, and 
language-minority issues are in short supply. 

b. Qualified bilingual teachers are in limited supply. 

c. Some staff are reluctant to work in the evening. 

d. Budgets are insufficient to support staff salaries. 

By identifying obstacles and discussing possible solutions, 
participants can focus on alternatives to consider for overcoming 
significant barriers to improving their projects. 

@ Identify resources needed to address each objective and obstacle. 

Staff need to identify the resources necessary to address objec- 
tives and obstacles. Resources include personnel, time, materials, and 
funding. This step helps staff analyze the scope of the project and 
establish realistic expectations for what can be accomplished in the 
project. 

O Assign staff members responsibility for designing and 
implementing each task or activity. 

When staff members are responsible for particular tasks, they 
become stakeholders who regard their contributions as important 
factors in the success of the project. Accountability is especially 
important in collaborative efforts to ensure that each team member 
contributes. 

O Establish a time line for each task or activity. 

Because family literacy projects offer a broad array of services 
and activities over an extended period of time, staff members need to 
specify a time line to ensure that all steps of the planning process arc 
implemented. Steps 6-8 are essential because they provide the param- 
eters for developing each activity. 

Examples of Steps 6-8. 

When staff members of the FELP project located at California 
State University, Sacramento ( 1986-89) identified resources needed to 
improve the project (Step 6), they determined that having their own 
curriculum would significantly facilitate the learners' achievement of 
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the project's objectives. During the first year of project implementa- 
tion, the staff compiled curriculum content and activities thai the 
learners especially liked. Each staff member assumed a specific role 
for designing, compiling, and evaluating the curriculum materials 
(Step 7). The adult learners also had responsibilities such as identify- 
ing content they believed was important to learn and giving feedback 
regarding the effectiveness of learning activities. After deciding on 
the scope of the curriculum, the staff created a time line for develop- 
ing the curriculum for the project (Step 8). 

During the second year, the staff field tested and refined the 
curriculum materials. By the end of the third year, the curriculum was 
printed and used at all site? in the project. Titled Parenting Curricu- 
lum for Language Minority Parents (Holt, 1988), the text was devel- 
oped in English, Chinese, Hmong, Khmer, Korean, Lao, Spanish, and 
Vietnamese. A teacher's activity guide also was developed. The 
curriculum is an example of how staff can collaborate with learners to 
identify and develop resources that enhance the implementation of the 
project. The success of the effort was dependent on ( I ) staff members' 
having an intimate knowledge of the adult learners* needs; (2) staff 
and adult learners' sharing responsibilities for tasks; and (3) staff 
members' adhering to a time line for completing the tasks. 

Parenting Curriculum for Language Minority Parents has also 
been a valuable resource for FELP project staff of the Stockton 
Unified School District (1988-91). Staff used the curriculum to 
improve parents* knowledge of such topics as parent-child communi- 
cation, child development, discipline techniques, and the U.S, educa- 
tional system. Each topic, however, was addressed only after the staff 
and learners agreed that it was important. 

Staff of the FELP project in the Ravenswood City Elementary 
School District (1989-92) found that computers were an important 
resource for enhancing the implementation of their project (Step 6). 
The computers were used to supplement the literacy instruction the 
parents received in Spanish and English. For example, beginning- 
level learners composed stories on the computer, and advanced learn- 
ers created their own resumes and developed a project newsletter. The 
computers became a resource for motivating parents to participate in 
project activities and for strengthening the relationship between par- 
ents and their children. 

@ Specify expected outcomes. 

During this step of the planning process, staff and learners iden- 
tify the type and amount of change or growth they expect at the end of 
a given lime period. This step represents what the project participants 
regard as success in the project. Growth can be expressed in terrus of 
academic growth (for example, a specified increase in positive ratings 
on a project-developed reading test); social, cognitive, or affective 
development; organizational skills; or parenting skills. The outcome 
- statements are more detailed than the objectives that were developed 
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earlier in the process (Step 3). They should specify the assessment 
measures to be used and the amount of growth ihat seems realistic. 
Staff members and learners may want to develop outcomes that are 
short term (e.g., the ability to explain one's needs to an apartment man- 
ager) and long term (e.g., obtaining a teaching job). 

(J) Design assessment instruments and procedures to collect data. 

Staff and adult learners need to identify the data that will docu- 
ment progress toward attaining the outcomes specified in Step 9. For 
example, taking a frequency count of the number of times children read 
to parents per month might be an indicator of parents' encouragement 
of children's literacy development. 

Outcomes may be assessed with various types of assessment 
strategies. For instance, an outcome such as growth in listening com- 
prehension could be assessed through a single measure such as a stan- 
dardized test, an interview, an observation measure, or performance 
sample. However, an outcome that deals with reading strategies that an 
adult learner uses to obtain information might be assessed through a 
combination of an interview, an observation measure, and a perfor- 
mance sample to fully capture all the strategies that the reader might be 
using. To gather data accurately, it is ir* portant that a match be estab- 
lished between the type(s) of assessment and the nature of the out- 
come. A variety of alternative procedures for assessing specific out- 
comes are discussed in Chapters HI, IV, and V. 

Analyze data. 

Data become meaningful when they are analyzed. Various ap- 
proaches can be used for data analysis. Staff may want to represent 
assessment data in numerical scores to generate percentages, means, 
and standard deviations. On the other hand, staff may wish to describe 
learner outcomes by summarizing assessment results in narrative form. 
To interpret and analyze data effectively, staff members need to under- 
stand thoroughly the outcomes ihat were assessed. Chapter V contains 
detailed suggestions for analyzing data. 

^) Formulate conclusions and implications. 

The results of the data analyses should be reviewed by staff and 
learners to determine the extent to which they accomplished what they 
sel out to do. That is, were the objectives of the program attained? 
Were the outcomes specified in the objectives realistic? Were they 
matched to the learners* needs? Should instructional activities be 
changed? Should objectives be modified? This step provides staff and 
learners with the opportunity to make what arc often hard choices 
about the project. What has been successful? What needs to be 
strengthened? What ought lo be eliminated? What needs to be 
changed? 

As shown in Figure I , the decisions made in Step 12 should be 
used to refine the activities associated with Steps 1, 2, 3, and others. 
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Conclusions about the outcomes of project activities should influence 
the development of objectives that will be pursued in the future. That is, 
the planning process model represents a cycle of interdependent steps. 
When any aspect of ihe program is changed, all of the related aspects 
should be reviewed to determine if they also need to be modified. Chap- 
ter V provides a variety of strategies for using and reporting data. 

Designing Alternative Approaches to Assessment 
and Evaluations Where Do You Begin? 

The preceding section presented a process for planning, imple- 
menting, and evaluating a family literacy project. The model empha- 
sizes the importance of collaboration among the staff members and 
learners for creating a successful project. The planning process also 
underscored the value of alternative assessment procedures for helping 
staff ascertain the needs of adult learners and the progress they are mak- 
ing in the project. 

The final section of this chapter presents suggestions for imple- 
menting the planning process model and related alternative approaches 
to assessment and evaluation. Implementing the model involves decid- 
ing ( 1 ) who will be involved in the planning process; (2) how the plan- 
ners will work together; and (3) what issues the planners will address in 
assessment and evaluation. 

Members of the Planning Team 

The first and perhaps most important step in initiating a compre- 
hensive planning process is to decide who will be involved in the pro- 
cess. A planning team made up of staff members and learners will help 
ensure that decisions are based on a thorough understanding of the 
needs of all participants. The project evaluator might also be included 
in the planning team to assist the staff and learners with designing the 
evaluation. 

To the extent possible, all instructional staff should have the op- 
portunity to participate in the planning proeess. Community liaisons 
also should participate because of their knowledge of the learners" lan- 
guage and cultural backgrounds. When deciding on the content of the 
curriculum and specifying learner outcomes, it may be helpful to in- 
clude suggestions from prospective employers, teachers of the partici- 
pants* children, representatives from local government agencies (for 
example, employment and social service offices), and staff from com- 
munity colleges and uni versities. 

Approaches to Planning 

After the planning team has been formed, team members need to 
consider how they can work together effectively. Because of the diver- 
sity of individuals and the range of perspectives that need to be blended 
into a consensus, it is vital that the facilitator of the planning team be 
both willing and able to promote a cooperative spirit among team mem- 
bers. The facilitator needs to ensure that each team member's contribu- 
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dons are invited, expressed, valued, and integrated into the deci- 
sions that are made in each step of the planning process. 

The facilitator has the responsibility of ensuring that meet- 
ings and tasks are structured so that the project benefits from the 
experiences of all team members. A collaborative approach to 
planning enables staff and learners to become moie knowledge- 
able about each other's needs and the value of cooperation itself. 
According to Freire (1973), "acquiring literacy does not involve 
memorizing sentences, words, or syllables — lifeless objects 
unconnected to an existential universe — but rather an attitude of 
creation and re-creation, a self -transformation producing a 
stance of intervention in one's context" (p. 48). Collaboration 
helps team members develop a sense of responsibility for help- 
ing others and the feeling of satisfaction that comes from a suc- 
cessful team effort. The act of planning and implementing to- 
gether can provide significant opportunities for learning how to 
assess what one needs and how to take steps to improve one's 
context, whether it be at home, at school, or at work. 

Issues in Assessment and Evaluation 

The primary purpose of the planning team is to take the 
necessary steps for designing, implementing, and evaluating the 
project. This section presents questions that the planning team 
needs to answer related to designing the assessment and evalua- 
tion of the project. 

1. What should be evaluated? 

Staff members need to consider how they will collect data 
for assessing the implementation of the project (process evalua- 
tion) and the effects of the project on the learners' lives (product 
evaluation). To design the process evaluation, the planning team 
should deal with such issues as the role of the staff members and 
learners in planning, implementing, and evaluating the project. 
Other points to examine might be the extent to which the goal s 
of the project arc consistent with the goals of the learners. The 
planning team needs to decide what data should be collected, 
analyzed, used, and reported to provide a comprehensive picture 
of the implementation of the project. 

Product evaluation, on the other hand, deals with issues 
related to the effects of the project on adult learners (and their 
children, if the objectives of the project include children). To 
design the product evaluation, the most important point for the 
planning team to address is the specification of what constitutes 
success in the piujeci. In other words, how will the staff and 
learners know the extent to which the goals and objectives of the 
project are being met? How will participants determine thai 
learners have increased their ability to use language for various 
purposes or their self-confidence in working with others? What 
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data will be used to make these determinations? How will the data be 
collected? 

To determine the value of collecting any piece of data, the plan- 
ning team needs to clarify the purpose of the data and analyze whether 
the cost of collection is worth the required time and resources. For 
example, will the data be used to improve initial assessment; to 
enhance the curriculum; to show learners the progress they are making; 
to develop staff members' teaching ability; to document progress for 
outside agencies; or to improve the administration of the project? 

2, When should data collection take place? 

After determining what data will be collected, the planning team 
needs to decide when the data will be gathered. Questions ;o address 
include the following: Are the data needed before the learners enroll in 
the project? How many times should data be collected to show 
progress? Chapters III and IV contain information about the timing of 
initial and ongoing assessment procedures. 

3, How will adult learners, staff members, the project evaluator, 
members of the community, and others be involved in conducting 
the evaluation? 

The planning process model emphasizes the importance of col- 
laboration in helping to ensure an evaluation design that is comprehen- 
sive and meaningful. The planning team needs to consider who should 
be involved in the evaluation, the role of each individual, and strategies 
for facilitating a cooperative effort. 

4, What data collection procedures will be used? 

The planning team should decide what procedures will be used to 
collect the data. For example, what data will be gathered via alternative 
instruments such as surveys, interviews, observation measures, and 
performance samples? How will standardized measures be used? Fac- 
tors to consider in these decisions are the comparative effectiveness of 
each procedure in providing accurate and reliable data; resources 
needed for developing the procedures; staff members' knowledge or 
prior experience in using the procedures; time and resources necessary 
for collecting and analyzing the data. 

5, How vcill staff members know that progress is being made? 

Objectives and expected outcomes of the project represent the 
staff members' criteria for project success. For most family literacy 
projects, criteria include a range of outcomes stich as gains in listening, 
speaking, reading, and writing abilities; improved self-confidence; 
enhanced use of literacy in different contexts; increased abilities of 
parents to help their children in school; improved involvement of par- 
ents in their children's school activities; and greater use of community 
resources. Chapter IV provides examples of other learner outcomes and 
related alternative assessment procedures. 
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6. How will stakeholders and others become knowledgeable about 
the results of the evaluation? 

One of the uses of the evaluation results is to inform stakeholders 
of the nature of the literacy project and the progress lhal the learners are 
making. Stakeholders may include the learners in the project, other staff 
members, administrators, family members, and representatives of the 
funding agency . The planning team should determine who needs to ie- 
ceive the evaluation results and how the results will be disseminated. 
Chapter V provides examples of strategies for reporting the results of 
the evaluation. 

Summary 

Traditional standardized assessment and evaluation procedures 
provide only some of the answers to the questions that staff and others 
may have about family literacy projects. The standards on which tradi- 
tional procedures are based are not always tailored to the needs, goals, 
and characteristics of the diverse adult learners who participate in lit- 
eracy projects. In contrast, alternative approaches to assessment and 
evaluation enable project staff to create instruments and procedures that 
provide highly useful and meaningful information for improving their 
project. When standardized and alternative assessment procedures are 
integrated, they can provide a more complete understanding of the lit- 
eracy project — its objectives, activities, and results — and help meet the 
information needs of the learners, program staff, and funding agency. 

To be effective, alternative approaches should be linked carefully 
to the needs of ihe adult learners, literacy framework, objectives, and 
curriculum content of the project. Such linkage can be achieved by 
using a planning model that is designed and implemented through the 
collaborative efforts of staff and learners. The collective commitment of 
all project participants to a comprehensive planning process will help 
ensure that assessment and evaluation activities are implemented as an 
integral part of planning and implementing the project. When assess- 
ment and evaluation are integrated with instruction, they produce 
results that can be used to document success and plan improvements in 
the project. 



45 



Chapter II 



References 



Auerbach, E. R. (1992). Making meaning, making change: Participa- 
tory curriculum development for adult ESL literacy. Washington, DC 
& McHenry, IL: Center for Applied Linguistics & Delta Systems. 

California Department of Education. (1992). Model standards for 
adult English-as-a-second-language programs. Sacramento: Author. 

Family English Literacy Program: ESEA Title VII application. (1991, 
November). Sacramento: Sacramento City Unified School District. 

Freire, P. (1973). Education for critical consciousness. New York: 
Seabury. 

Holt, G. D. (1988). Parenting curriculum for language minority 
parents. Sacramento: California State University, Cross Cultural 
Resource Center. 

Johnson, D. W. (1987). Human relations and your career: A guide to 
interpersonal skills (2nd ed.). Englewood Cliffs, NJ: Prentice-Hall. 

Lytle, S. L., Belzer, A., Schultz, K., & Vannozzi, M (1989). Learner- 
centered literacy assessment: An evolving process. In A. Fingeret & 
P. Jurmo (Eds.), Participatory literacy education (pp. 53-64). San 
Francisco: Jossey-Bass. 

Lytle, S., Marmor, T., & Penner, F. (1986, April). Literacy theory in 
practice: Assessing reading and writing of low-literate adults. Paper 
presented at the meeting of the American Educational Research 
Association, San Francisco, CA. 

Proceedings from the Family English Literacy Seminar. (1991, 
January). Sacramento: California Department of Education, Bilingual 
Education Office. (Available from the National Clearinghouse for 
Bilingual Education, Washington, DC.) 

Results of the focus group meeting: Alternative approaches to 
evaluating Family English Literacy Programs. (1991, January). 
Sacramento: California Department of Education, Bilingual Educa- 
tion Office. 

Scribner, S. (1987). Introduction to theoretical perspectives on 
comparative literacy. In D, A. Wagner (Ed,), The future of literacy in 
a changing world (pp. 19-24). New York: Pergamon. 

Wolf, D., Bixby, J., Glenn, J., Ill, & Gardner, H. ( 1 99 1 ). To use their 
minds well: Investigating new forms of student assessment. Review 
of Research in Education, 17, 31-74. 

Wrigley, H. S., & Guth, G. J. A. ( J992). Bringing literacy to life: 
Issues and options in adult ESL literacy. San Mateo, CA: Aguirre 
International. 



Assessing Success 

4fi 



BEST COPY AVAILABLE 



QSSESSING 0VCCESS 

CHAPTER III 

INITIAL 
ASSESSMENT: 
FIRST 
STEP 
TO 
SUCCESS 




Kathy Graham 
Lincoln Unified School District 
Stockton, California 



Assessment* plays an important role in the successful operation of family 
literacy projects — from initial planning to the final stages of implementa- 
tion. Initial assessment is pivotal in determining the links between instruc- 
tional services and learners' needs. This chapter explores the purposes of 
initial assessment and examines a variety of methods and instruments that 
can be used to accomplish these purposes. The information in this chapter is 
divided into two sections. The first section focuses on initial assessment as 
an integral part of the planning of a successful family literacy project. The 
second section describes the many uses of initial assessment in the imple- 
mentation and evaluation of a project. 



■ Sec Ihc Glossary on pages 1.17-1 Mi ol ihis puhhcaiion fur dcrimlioi. - ,tuii< t:td key terms that 
appear m tbw chap lor. 
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Initial Assessment in 
Planning a Successful Project 

This section focuses on initial assessment as an integral part of 
the planning of a family literacy project — Step 2 in the Planning 
Process Model introduced in Chapter 11. The following issues will be 
addressed: (I ) the importance and nature of initial assessment in the 
planning of a family literacy project and (2) approaches that can be 
used for initial assessment 

Importance and Nature of Initial Assessment 

Initial assessment is important in the development of a success- 
ful literacy project because it provides program planners with the 
opportunity to examine the needs, desires, and goals of the potential 
participants. This information can be used to generate the instruc- 
tional objectives and other aspects of the program design. Figure I 
illustrates how. the results of a needs assessment influence the key 
aspects of the project. 

An effective needs assessment can help staff understand the 
needs and goals of the participants and establish the foundation for the 
entire project. Objectives are derived from the identified needs; 
activities are designed to achieve the objectives; and assessment 
procedures are created to evaluate attainment of the objectives. (The 
effects of initial assessment are illustrated in more detail in "Planning 
Process Model," Figure 1 in Chapter 11 of this volume.) 



Figure 1 
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A family literacy project should be planned on the basis of 
extensive information obtained about trie learners. According to 
Woinstein-Shr (1993), project staff may wish to learn about ( 1 ) the 
structure and characteristics of the community in which the learners 
live; ( 2) the language, literacy, and educational background of the 
learners; and (3) the concerns of adults in their role as parents. The 
following list expands these issues and presents a wide range of 
information for staff to consider in designing a needs assessment: 

1 . Demographic information, about the learners' community (for example, 
number of language groups, length of residency, and degree of tran- 
siency). 

2. Historical and cultural backgrounds of the learners (for example, country 
of origin, obstacles faced in corning to the United States, family structure, 
and religious affiliation). 

3. Information about family members (for example, average number of 
family members, socioeconomic level, and literacy levels in English and 
their native language). 

4. Educational, parenting, career, psychosocial, and health needs of the 
parents (for example, educational background, reasons for learning 
English, educational expectations for children, vocational goals, and 
support network). 

5. Educational, psychosocial, and health needs of the children (for example, 
educational background, language proficiency, and health history). 

6. Determination of which needs are short term and which are long term (for 
example, the learners' psychological needs that require immediate 
attention versus their interest in attending college). 

Other needs may be identified by consulting with teachers, adminis- 
trators, service providers, and others in the community. 

Approaches for Initial Assessment 

Initial assessment to determine the needs of potential project 
participants may involve the use of surveys, interviews, and focus 
group discussions with community members as well as analysis of 
documents and literature regarding family literacy. Each of these 
approaches is described below, with examples of how they were 
carried out for a needs assessment in the Lincoln Unified School 
District in Stockton, California. An in-depth explanation of the 
process of analyzing and reporting the results of these approaches is 
given in Chapter V. The use of standardized tests, such as the Basic 
English Skills Test (BEST) (Center for Applied Linguistics, 1982), in a 
family literacy project is addressed in Chapter VI. 

Surveys. Surveys are effective for assessing the needs of 
potential project participants. The survey may consist of a written 
questionnaire or cheek list that is completed by prospective partici- 
pants or administered orally by bilingual staff members to individuals 
in English or their native language. Responses to closed-ended 
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questions may be easier to tally, but open-ended questions can give 
staff a more complete understanding of the learners 1 needs. At the 
beginning of the planning process, a survey can be administered to a 
large group of individuals to create a general picture of the needs of 
adults and other family members who may be interested in project 
services. 

For example, in 1990 the staff of the Lincoln Unified School 
Districi developed a survey shown in Example I (see page 48) to 
collect basic information Irom prospective participants regarding their 
need for a family literacy project. The survey addresses such issues as 
number of adults and children residing in each housing unit; their 
native language; number of adults who would like additional English 
instruction; preferred location and time of literacy classes; parents' 
need for child care and transportation; desired curriculum topics; self- 
assessment of English language proficiency; number of years' resi- 
dence in the United States; and number of children who are enrolled in 
ESEA Title VII projects. To obtain more detailed information about 
what the participants would like to learn in the project, staff could use 
a survey such as the one shown in Example 2 (see page 49). 

To get the most comprehensive and valid results, four bilingual 
staff members of the district visited 300 apartment units during a two- 
week period to administer the survey orally to individual residents. 
This instrument, and others shown in the chapter, can be administered 
in English or the participants' native language. The survey results 
indicated that the Cambodian adults living in the targeted area were 
especially interested in services that a family literacy project could 
provide. For eample, parents wanted to know how they could help 
their children learn Khmer and how they could volunteer in the 
schools. Staff also realized that the parents 1 limited access to transpor- 
tation made it necessary to have literacy classes located near their 
apartments. On the basis of those findings, the staff decided to estab- 
lish a family education center in a vacant apartment located in the 
middle of a two-block area of five apartment complexes occupied by 
Cambodian families. The survey also helped staff determine the most 
convenient times for literacy classes, the literacy needs of the prospec- 
tive participants, and their related needs, such as counseling and 
cultural adaptation. 

A door-to-door survey such as this one is practical when the 
prospective participants live in a concentrated area. If families are 
dispersed, bilingual staff can administer the survey via telephone. 
Because the Vietnamese families live throughout the Lincoln district, 
bilingual staff used the telephone to administer an adapted version of 
the survey shown in Example 1 . The needs of the Vietnamese families 
differed significantly from those of the Cambodians. Rather than 
attending literacy classes in their neighborhood, the Vietnamese 
adulls preferred to go to one of the district's school sites for classes. 
They also indicated their desire for citizenship classes, whereas the 
Cambodian adults did not express this interest, a difference that 
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became understandable in light of other kinds of information 
gathered during the needs assessmeni process. For example, 
almost all the Cambodians surveyed were forced to leave their 
country by the Communist Khmer Rouge and looked forward to 
returning in the future, whereas most of the Vietnamese adults 
chose to come to the United States for political freedom and 
educational opportunity. These major differences between the 
two community groups highlight the crucial importance of 
conducting a needs assessment when planning a project. 

Focus groups. Focus group discussions help staff mem- 
bers gather information from a large number of people in a short 
amount of time. A focus group discussion is conducted with 
small groups according to a protocol that focuses on specific 
topics, procedures, and questions. The goal of analyzing focus 
group data is to examine systematically the participants' 
comments to identify trends or patterns that recur among 
different individuals or groups. As pare of the initial needs 
assessment for its family literacy project, staff of the Lincoln 
Unified School District conducted locus group discussions with 
members of the district's bilingual advisory council; potential 
project participants; the participants' children; and representa- 
tives of the school district, including teachers and 
administrators. 

Interviews, interviews are effective for obtaining detailed 
information from individuals. Lincoln's staff interviewed 
service providers in the community, such as librarians, and 
visited family literacy projects located in northern California to 
determine how the literacy and social needs of Southeast Asian 
families were being addressed by various service providers and 
the relative effectiveness of those services. The interviews 
provided yet another view of the learners' needs and an under- 
standing of the level of assistance available to meet the needs. 
The information also helped identify content and methodology 
for literacy classes and strategies for coordinating project 
services with other community agencies. 

Document and literature review. Learners' needs also can 
be assessed by analyzing documents and literature related to 
family literacy. Data can be gathered from school district 
records, including test scores, gnidc*, and attendance records of 
participants 1 children; welfare department statistics such as the 
socioeconomic status of participants; and journals, newsletters, 
books, and state and national reports that include information 
about the literacy -related need? of participants. 

Staff in the Lincoln district analyzed demographic data 
and statistical reports regarding Southeast Asian families. For 
example, staff identified the number of Cambodian and other 
Southeast Asian families living in San Joaquin County and then 
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projected the number that would be residing in the district in the future. 
In addition, they consulted the Report to the Congress: Refugee Re- 
settlement Program (U.S. Department of Health and Human Services, 
1 989), * vhich indicated that the reading and employment levels of 
Camboc.ian adults in the district were significantly below the national 
averages for other Southeast Asians. 

The combined findings from surveys, focus groups, interviews, 
and research reports constitute a thorough assessment of the need for a 
family literacy project. After gathering the information, project staff 
should analyze results, identify commonalities, and develop a compre- 
hensive summary that reflects all the needs of the targeted participants. 
The next step is for staff to select those needs that will be the focus of 
the project and that can realistically be met with available resources. 
Funding agencies recognize that not all needs can be met by any one 
project and that the scope of a grant proposal should be limited and 
elear. 

A thorough needs assessment enables staff to address the next 
steps of the planning process: that is, (1 ) developing detailed program 
objectives that respond to the needs; (2) identifying the curriculum and 
instructional activities that will lead to the achievement of each objec- 
tive; and (3) developing assessment procedures to measure the achieve- 
ment of each objective and generate data for a final evaluation report 
(see Chapters I and II for more information on developing project 
objectives). Before moving on to these steps, however, staff and pro- 
spective learners should develop a consensus regarding the philosophi- 
cal basis of instructional methods, curriculum content, and other aspects 
of the project. Time should be devoted to the development of the 
project's literacy framework. 

Developing a Literacy Framework 

Staff members in many family literacy projects have found it 
helpful to develop a shared understanding among staff and Seamers 
regarding key aspects of the project. This shared understanding is 
referred to in Chapter II as the project's "literacy framework. " Such a 
framework will facilitate discussions and decisions about the needs 
assessment* program design, and evaluation. Issues to discuss include 
{ 1 ) learners' literacy needs; (2) goals and objectives of the projeet; (3) 
curriculum content and instructional methodology; and (4) approaches 
for assessing and evaluating the learners' literacy development. Devel- 
oping consensus on such complex issues is a challenging task, and staff 
and learners may vary in the extent to which they are able or willing to 
collaborate to reach common ground. It is important, however, for all 
participants to make an ongoing commitment to work together and to 
draw on each other's expertise. 

As general issues become resolved, staff and learners can pursue 
more detailed aspects of the project, such as the following: 
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1 . Role of literacy in the learners' daily lives. 

2. Relative importance of the goals and objectives of the project. 

3. Curriculum content and instructional methodologies, including the use of 
the learners' native language and English in the project, 

4. General design of the assessment and evaluation procedures. 

5. Strategies for ensuring that the project will be dynamic, flexible, and 
responsive to the learners' needs. 

It is important for the staff to validate the learners* native lan- 
guage within the literacy framework. Staff members need to support 
the learners' continued use and development of their native language 
for improving their literacy and strengthening their family. The 
learners' literacy experiences in their native language should also be 
analyzed and considered when designing the instructional program. 
Standardized proficiency and achievement tests are available in 
various languages and may be helpful in assessing the participants' 
nativ : language proficiency and educational background. Alternative 
approaches such as a writing sample may also be useful. Depending 
on the learners* needs, family literacy projects may provide ( I ) 
opportunities for further literacy development in their native lan- 
guage; (2) activities in which learners acquire challenging content 
through their native language; and (3) strategies for using learners' 
native language to support their children's development. A native 
language literacy program is especially helpful in giving learners 
successful schooling experiences before they have to adjust to an all- 
English environment. Many literacy projects also report that instruc- 
tion in native language literacy facilitates the learners' acquisition of 
English and enhanced self-esteem (E. P. Garza, personal communica- 
tion, May 14, 1993). 

An Expanded View of Literacy 

Any assessment of literacy-related needs should address the 
social contexts in which language and literacy are used. Literacy 
occurs everywhere in the adult learner's world: at home, in the work 
place, at school, and throughout the community. Furthermore, the very 
nature of literacy goes beyond communicating meaning through 
speaking, reading, and writing. As suggested by Eisner ( 1 99 1), 
literacy can be considered "the ability to encode or decode meaning in 
any of the forms used in a culture to represent meaning" (p. 1 1 ). 
Language, according to Eisner, is just one form of literacy; meaning is 
also conveyed through music, dance, and visual forms. 

The importance of Eisner's expanded paradigm of literacy 
became apparent to Lincoln Unified^ staff when the results of the 
needs assessment survey (Example 1) revealed that although the 
majority of the Cambodian participants never attended school in their 
native country and could not read or write in their native language 
(Khmer), they were experts at communicating meaning through other 
media, such as painting, music, classical dance, and wood carving, By 
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recognizing the participants' lifelong achievements and their unique 
levels of literacy, staff were able to create a program design that 
included activities to enhance the learners' self-esteem and build on 
their abilities. For instance, their program design included activities in 
which adult learners shared their areas of expertise (for example, 
nalive cuisine, folk dance, music, and sewing) with others. In develop- 
ing ihe instructional program, it is important to build on whai learners 
are already doing with literacy. A successful needs assessment will 
reveal learners* successes with literacy as well as their needs for 
expanding their uses of literacy. 

Initial Assessment in Implementing 
a Successful Project 

This section will describe the integral role of assessment in 
successfully implementing the project and refining the objectives, 
curriculum content, and instructional activities. Initial assessment is 
important in the implementation of any project because it allows 
project staff to ( I) reshape and refine their literacy framework; (2) 
obtain useful information about the learners during initial intake; and 
(3) establish important baseline data for assessment and evaluation. 

Refining the Literacy Framework 

A project's lUeracy framework should be synchronized with the 
needs, goals, past experiences, and cultural values of the participants. 
To ensure the relevance and effectiveness of the literacy framework, 
the staff can use survey results to guide them in the early stages of 
implementing the project. For example, the results of surveys can be 
used to identify curriculum content and align project goals and 
objectives with learners' needs, interests, and past literacy experi- 
ences. Administering surveys also has the effect of publicizing the 
availability of literacy classes and attracting learners to the project. 

During the initial stages of implementing the family literacy 
project at Lincoln Unified, staff visited individual apartments to 
administer a parent survey (Example 3, sec pages 50-5 1 ) orally in 
Khmer to prospective participants. This needs assessment was a 
follow-up to the survey conducted with Example 1. In addition to 
providing important data about the participants and the community, 
the survey enabled staff members to develop an in-depth awareness of 
the personal and educational needs and goals of the Cambodian 
families. The parent survey contains open-ended questions designed 
to elicit comprehensive responses. Administration of the survey 
helped staff and learners begin to build a bond of mutual trust and 
respect that is essential to the success of a family literacy project. 

Another way to maintain the relevance and effectiveness of the 
literacy framework is to conduct focus groups with learners at the 
beginning of the project and at regular intervals throughout implemen- 
tation. During the focus group activities, staff can check the learners" 
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own assessment of their progress and obtain their inpul regarding the 
relevance of the instructional program. 

Establishing an Intake Assessment Procedure 

The second purpose of initial assessment during the implementa- 
tion of a project is to provide the staff with intake data useful for 
several purposes: meeting the needs of individual learners, developing 
appropriate curriculum content and instructional methodology, and 
placing learners in appropriate instructional contexts. During this 
phase of initial assessment, staff can collect background information 
on actual project participants, including proficiency in English and 
their native language, previous educational experiences, and length of 
residency in the United States. Instruments for collecting basic intake 
information are shown in Examples 4 and 5 (see pages 52-54). 

Because they are often the learners' first in-depth experiences 
with the project, intake procedures become part of the foundation for 
establishing a good working relationship between staff and learners. 
Therefore, intake procedures need to give learners a positive intro- 
duction to the project and provide staff with useful information for 
planning instructional activities. Staff members can help learners feel 
comfortable during intake by employing appropriate cross-cultural 
strategies and using the learners' native languages. 

Most family literacy projects no longer rely solely on standard- 
ized tests for obtaining intake assessment data. Alternative ap- 
proaches, such as interviews and writing samples, can provide staff 
with meaningful, highly personalized insight into the learner's back- 
ground. The interview is a versatile assessment approach for obtaining 
many kinds of information and can range in length from a five-minute 
informal exchange to an elaborate two-hour session consisting of "an 
array of questions and activities designed to engage adults in self- 
exploration of their literacy practices and abilities" (Lylle, Bclzer, 
Schultz, & Vannozzi, 1989, p. 59). 

Aucrbach ( 1992. pp. 1 17-1 8) developed an abbreviated list of 
Lytle et al.'s interview questions according to categories of informa- 
tion about prospective learners. The following are some examples 
from each category: 

Students ' background 

• Where are you from? 

• What is your native language? 

• Do you have family here? 

Employ mem 

• Did you work in your native country ? What kind of work 
did you do ? 

• Do you work here? 
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• Do you do work that you are not paid for (for example, child care 
and community service}? 

Education 

• Did you go to school in your native country? 

• Do most people in your native country go to school? 

• Did your parents go to school? 

Conceptions about literacy 

• Do most people in your native country know how to read and 
write? 

• How is reading taught ? 
Reading 

• Do you like to read? Why? Why not? 

• Do you read at home? What do you read? When do you read? 

• What languages do you read? 

Writing 

• Do you like to write? Why? Why not? 

• Do you write at home? What do yGu write? When do you write? 

• What language or languages do you write? 

Support systems 

• What do you do when you have trouble reading or writing 
something? 

• Does anyone help you? Who? 

• Do yoa help anyone with reading and writing? Who? 

Auerbach emphasizes that in order to make the interviewee feel 
as comfortable as possible, the interviewer should be ready to conduct 
the interview in the native language if necessary. Furthermore, to 
create a nonthreatening atmosphere for prospective learners, inter- 
viewers should avoid taking notes during the interview. Important 
information or summary comments should be recorded after the 
interview, Interviewers may need to be trained in the use of effective 
interpersonal communication techniques so that a positive climate is 
established for the interview. Chapter V includes more information on 
the effective use of interviews. 

Lytle et al. (1989) and Auerbach (1992; suggest that, in addition 
to direct questions, interviews may include opportunities for learners 
to explore their abilities to read and write in their native language and 
in English. A reading sample might be used during the interview to 
assess the learner's reading ability. For example, an adult learner 
could be presented with a variety of reading materials in both lan- 
guages: newspapers, story books, labels from food, greeting cards, 
rental agreements, letters from school, or prescription labels. Learners 
then could be asked to select items they would like to read while staff 
assess their reading ability. For an assessment of writing ability in 
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their native language or in English, learners might be asked to produce 
writing samples as shown in Example 6 (see pages 55-56). The writing 
samples may be scored holistically because the staff is interested at this 
point in assessing the learner's overall writing ability. Holistic scoring 
yields' a single numeric value (usually between one and six) given to a 
writing sample as a whole. Chapter IV contains other suggestions and 
instruments for assessing reading and writing skills. 

Another practical approach to intake assessment is to use some 
lime during initial instructional activities to obtain information from 
learners that can help staff members make decisions about placement, 
curriculum content, and instructional activities (Auerbach, 1992). For 
example, during an introductory lesson, staff could include a writing 
exercise in which learners are given such writing prompts as these: 
Describe a favorite place in your country. Describe a favorite food. 
Explain why you are happy (or not happy) about living in the United 
States. The responses could provide staff with valuable information 
about the learners' previous experiences, current needs, and language 
abilities. 

Information obtained from writing samples can help staff mem- 
bers develop insight into learners' initial writing ability and place them 
in appropriate instructional settings. A series of at least three writing 
samples is useful for determining where to place learners in a multi- 
level program. Staff may want to ask participants to write about 
selected topics in their native language as well as in English. The 
samples can be kept in a portfolio for comparison with learners' work 
later in the project. For learners who have not yet learned to write, staff 
members can transcribe stories ft iat learners tell to staff members or 
other learners in their native language or in English. 

Reading proficiency also can be assessed during regular class 
activities. Project staff can provide various reading materials in the 
learners' native language and in English, such as stories, newspapers, 
poems, recipes, and personal letters. Learners can then choose an item 
and read it to the class. For more in-depth assessment, staff members 
can use a similar procedure with an individual learner while other staff 
members work on different activities with the rest of the class. 

Collecting Baseline Data 

The third purpose for using initial assessment procedures in the 
implementation stage of a project is to collect baseline data thai 
establish a starting point regarding a learner's literacy development, 
parenting experience, knowledge of community resources, self-esteem, 
confidence, interaction with children, and other outcomes that the staff 
would like to assess. Some family literacy projects use standardized 
tests to establish baseline data. If such formal assessments are used, 
learners may need a few days to feel comfortable about being in the 
project before they are tested. 

Alternative approaches such as surveys, interviews, and reading 
and writing samples, are especially useful for establishing baseline 
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data because they can be designed according to the objectives and 
curriculum uf die project. Samples of instruments that can be used as 
pre- and posttests are shown in Examples 7 and 8 (see pages 57-59). 
These instruments can be administered to learners in English as well 
as in their native languages. 

Summary 

Assessment and evaluation are not procedures tagged onto the 
responsibilities of project staff members simply to satisfy the interests 
of a funding agency or other stakeholders. Rather, they are important 
strategies for identifying learners' needs and abilities at the initial 
stages of a family literacy project and for ensuring the successful 
planning and implementation of the project. Sensitivity to and aware- 
ness of the needs, literacy levels, goals, and attitudes of students, 
whether they are adult learners, adolescents, or children, are essential 
if learning and personal growth are to occur. 

The project evaluator is an important contributor lo the assess- 
ment process. The evaluator needs to work closely with project staff 
to develop instruments and procedures for initial assessment. The 
evaluator can be helpful in designing instruments that are valid and 
reliable and are linked to project objectives. 

Chapter IV of this document provides a detailed description of 
instruments and procedures for assessing ongoing progress, and 
Chapter V explains how the results of initial and ongoing assessment 
procedures can be collected, analyzed, used, and reported for various 
purposes. Annual evaluation reports that combine the results of 
formal and informal, qualitative and quantitative, and authentic and 
standardized measures will provide a comprehensive, credible picture 
of the project-level success achieved by all participants in a family 
literacy project. 
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Example© 



NEEDS ASSESSMENT SURVEY 



Name 
Address 



Telephone ( ) 



Naiive Language 



1 

1 . 


How many adults live in your apartment? 


1. 


2 


How many children in your apartment attend Lincoln Unified School District? 






{Identify individual schools by name) 


z. 


3. 


How many adults currently attend GAIN program English classes? 


1. 


4. 


Where do they attend classes? (e.g., E)elta College, adult school) 


4. 


5. 


If there are adults living in this apartment who are not attending GAIN English 






classes, would they attend GAIN English classes if they were held in your 






apartment complex? 


5. 


6. 


How many men would go? 


6. 


7. 


How many women would go? 


7. 


8. 


Would these adults agree to attend class four hours a day, five days a week? 


8. 


9. 


Would they need to have someone outside the family take care of their 






children during class ? 


9. 


10. 


How many children need child care? 


10. 


11. 


Should class be held in the morning or in the afternoon' 1 


11. 


12. 


Is it a good idea for us to try to get our classes approved for GAIN, or should 






we have our own literacy program? 


12. 


13. 


If we cannot get our English classes approved for GAIN, would you attend 






anyway? 


13. 


14. 


How many hours per day should classes be held? 


14. 


15. 


How many days per week should classes be held ? 


15. 


16. 


What would you like to learn? 


16. 




a. English d. How the schools operate 






b. How to get a job e. How to become a citizen 






c. How to be a good parent 




17. 


What is your level of English language sitills? 






a. Oral skills: ( 1 ) Low (2) Medium (3) High 


17a. 




b. Reading and writing skills: ( I ) Low (2) Medium (3) High 


17b. 


18. 


How long have you been in the United States? 


18. 


19. 


How many children in your apartment are participating in ESEA Title VII programs? 


19. 
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Example Q 

OMS-FELP NEEDS ASSESSMENT FOR PARTICIPANTS 



English 

Oral Skills: Where do you want to use better English ? 

grocery store children's school 

department store telephone 

doctor's office library 

drugstore immigration office 

bank welfare office 

post office neighbor/friend's home 



social security office 
apartment office (manager) 
restaurant 

flea/farrner's market 

temple/church 

other 



Reading Skills- What do you want to read better? 

school reports 

_ labels 

forms newspapers 

driver's manual schedules (bus, train) 

income tax forms advertisements 



signs 

bills 



school letters 
business letters 
checks (from jobs) 
job applications 
other 



Writing Skills: What do you want to write better? 

notes to children's schools fill out forms 

schedules for children business letters 



checks 
other 



Parenting: What do you think is important to learn in a parenting class? 

how to discipline children 

how to talk/share feelings with children 

how to help children feel good about themselves and their culture 

how to help children adjust at school 

good nutrition 

health (nutrition, personal hygiene, healthy habits) 

making the home and play areas safe 

monitoring home activities, including the use of television 

other 



Family Literacy: What do you want to do more of to promote literacy at home? 

ask children to read to you talk to children's teachers 

take children to the library encourage children to write 

tell stories to children visit children's school 

read to children other 

encourage children to do homework 
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PARENT SURVEY 



Name 
Address 

Native language 

Can you read in your native language? 



Telephone 
Native country 

Can you write in your native language? 



Year of" arrival in U.S. 


Single mother Single father Married 


Name of Children 


_ Grade Name of School 


1.. 




2.. 




3., 




4.. 




5.. 




6-. 




7.. 




8.. 




9.^ 




10. 





Are you presently attending a GAIN class? Yes No Where? 

What time is best for you to attend FELP classes'? morning afternoon evenings 
What days are best for you to attend class? 

Monday Tuesday Wednesday Thursday Friday 

What do you need to help you attend classes? Child care Translator 

Friend to come with you Other 

What would you like to learn? 

Apartment management (how to work with the apartment manager, get something repaired, ask about 
rent, look for an apartment) 

Citizenship (how to become a U.S. citizen) 

Communication with legal authorities (attorneys, police, legal support, legal aid, neighborhood watch) 

Education (registration, how to notify the school of illness* talk to the teacher/principal, visit your 
child's school, understand grades and progress reports) 

English as a second language (oral communication, reading, and writing) 
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Parent Survey (continued) 



Health (immunizations, hygiene, medical and dental check-ups, how to talk to (he doctor and dentist, 
fill out medical forms) 

Mathematics (money, banking, numbers, addition, subtraction, multiplication, division) 
Nutrition (common foods in the United States, healthful eating) 

Parenting/child rearing (developmental stages, suggestions on how to help your child at home, how to 
communicate effectively with your child) 

Safety (emergencies, car seats, fire prevention, water safety, first aid) 

Other 



6. What kind of help do your children need? 
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Example Q 
APPLICATION FOR FELP CLASSES 



NameiLiinj 
Address iSireeu 
Telephone number ( 
Native country 



(Middle) 



(Cuy) 

Birth date 

Date of arrival in U.S. 



(State) (ZIP Code) 

Male Female 



Did you go to school in your country? Yes No How long? 

Do you read in your native language? Limited Fluent 
Do you write in your native language? Limited Fluent 
Whai other languages do you speak? 



What other languages do you read? 



Whal other languages do you write? 



Have you taken English classes before? Yes 
Where? 

Are you; Married 

Single 
Name of husband/wife? 
Number of children Ages 
Health problems 



No How long? 

Divorced 
Separated 



Widowed 



Social Security number 
Alien registration number 



Office use only — Please do not write below this line. 
Date application received: 
Dale of test/s: 
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Example Q 
REGISTRATION FORM 

(To be completed by staff about family members participating in the project.) 

Family name 
Father' s name 
Mother's name 



PllM 

Firsi 



Last 
Last 



Name of Children 



Name of School 



Grade/Room No. 



1. 
2. 

4. 
5. 
6. 
7. 
8. 
9. 
10. 



Address 

What country do you come from? 

How long have you been in the United States? 

What level of education 
did your father complete? 

Did your children go to school in your country? 

Father's work 



Telephone ( ) 



How long have you been in Solana Beach? 

What level of education 
did your mother complete? 



Mother's work 



Arc you enrolled in any other ESL class? Yes No 

Do you have books at home? Yes No 

How many books do you have at home? 1-10 1 1-20 
Are they in English or Spanish? English Spanish 

Do you have a library card? Yes No 

Do you have a television in your home? Yes No 



21-39 
Both 



More than 40 
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Registration Form (continued) 

Do you watch television programs in English or Spanish? English Spanish Both 

Do you listen to radio programs in English or Spanish? English Spanish Both 

Do you read newspapers in English or Spanish? English Spanish Both 

Do you correspond with your family in Mexico? Yes No 
Does the father: 



Understand spoken English? Yes 


No 


Speak English? 


Yes 


INO 




Read English? 


Yes 


INO 




Write English? 


Yes 






Read Spanish? 


Yes 






Write Spanish? 


Yes 


Nn 




Does the mother: 








Understand spoken English' 


i es 


No 


Speak English? 


Yes 


No 




Read English? 


Yes 


No 




Write English ? 


Yes 


No 




Read Spanish? 


Yes 


No 




Write Spanish? 


Yes 


No 




Do the children: 








Understand spoken English 


? Yes 


No 


Speak English? 


Yes 


No 




Read English? 


Yes 


No 




Write English? 


Yes 


No 




Read Spanish? 


Yes 


NO 




Write Spanish? 


Yes 


No 




What problems have you encountered in the United States that you would like to discuss- in class? 
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Example© 
INTAKE ASSESSMENT FORM 



Name (Last) 

Write numbers 1-27, 



<RrsU 



(Age) 



27 



Wriie the alphabet. 

A 



Read ihe sentences and circle the correct word In the box 

1. He go at went to work. 

2. She am is are thirty-five. 

3. How old am is are you? 

4. I go going went to buy food yesterday. 

5. We came come came from to the United States five years ago. 



6. They has have got five children. 



7. I has had have been studying English for six months. 
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PRIMARY LANGUAGE WRITING SAMPLE 





Write about the picture in your native language. 
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Example @ 

PRE/POST TEST: COOKING AND NUTRITION 



Circle the foods that are healthy snacks. 




candy apple doughnut peanuts soda 




Circle the correct answer. 



1 . What's this? 


2. What's this? 


3. What's this? 


4. What's this? 














doughnut 


cake 


cheese 


hamburger 


pie 


cookies 


sandwich 


salad 


hamburger 


ho* dog 


meatioaf 


pizza 



Make an X on the foods that are not healthy snacks. 




Adapted from Holt. Q 0 < 1968) Parenting cvmculum for language mtnonty parents. Sacramento: California State University, Cross 
Cultural Resource Center 
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Example© 

PRE/POST TEST: THE U.S. SCHOOL SYSTEM 



•It 



Circle the one correct answer for each of the questions below, 

1 . The report a school sends home to tell about a student's progress in school is called a 

A. Cinch notice 

B. Emergency card 
C Report card 

D. Absence report 

2. The top person in an elementary school is called a 

A. Librarian 

B. Principal 

C. Teacher 

D. Vice-principal 

3. High school ends at grade 

A. Six 

B. One 

C. Nine 

D. Twelve 

4. A student checks oul books from the 

A. Cafeteria 

B. Plavground 

C. Office 

D. Library 

5. When a child is too sick to go to school he or she should 

A. Stay home and play outside 

B. Rest inside the house 

C. Watch TV all day 

D. None of these 

6. What things are needed for a child when leistering for school? 

A. Birth certificate 

B. Shot records 

C. Physical examination 

D. All of the above 

Answer the following questions true (yes) or false (no). 

7. Children need more sleep than adults. _ 

8. A parent eaunol he!p a child in schoul if the parent does not speak English. _ 

9. Schools do not want the parents lo help their children wiih school work. _ 

10. Teenagers go to high school. 



Assessing Success 



70 



Pre/Post Test: The U.S. School System (continued) 

If you have a child in school, answer the following questions. 

12. What grades are your children in? 



1 3. Whai schools do your children attend? 



14. Where are your children's schools located? 



15. What are the names of your children's teachers ? 
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CHAPTER IV 

ASSESSING 
PROGRESS: 
ARE WE 
PROGRESSING? 



To judge how well a family literacy project* is working on a day-to-day 
basis, staff members need to lake the educational pulse of the program 
periodically and assess learners' progress. Progress assessment can help 
staff determine the extent to which their instructional efforts lead to benefi- 
cial results and have positive effects on the learners' lives. Assessments also 
give participants a sense of accomplishment by showing them that they are 
making headway toward attaining their goals ( Wvigley, in press). To be 
effective, progress assessments need to include several components: 

1 . Baseline daia that show what participants know and what they can do. 

2. Indicators of progress or descriptors of what counts as success. 

3. Methods to gather information on a continuous basis. 

4. Plans for synthesizing and analyzing the data that are collected. 



• See the Glossary on pages 1.V7- 1 1H i>f this publication for definitions of Uahrizrd key terms thai 
appear in tins chapter 




Heide Spruck Wrigley 
Southport Institute for Policy Analysis 
Washington, DC 
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Alternative 
assessments can focus on 
the way learners use language 
and literacy as opposed to how 
much they know about 
grammar, phonics, or 
spelling. 



This chapter shows how staff can develop progress indicators based on 
the objectives of their project (see Chapters I and II for more 
on objectives). Six sets of instructional activities are presented, each 
of which focuses on particular learner outcomes and provides ex- 
amples of alternative approaches for assessing the learners' progress 
toward the outcomes. These activities include the following: ( 1 ) 
helping learners and staff get acquainted; (2) linking project activities 
with family life; (3) reading to do; (4) writing to do; (5) building self- 
esteem and taking literacy beyond the classroom walls; and (6) 
developing literacy through language experience. 

This chapter illustrates how learners' performances in these 
activities can be assessed with a variety of approaches such as obser- 
vation measures, interviews, and performance samples. The examples 
of alternative assessment instruments presented in this chapter are not 
intended as ready-to-use assessment tools. Staff members should 
adapt the instruments and procedures based on the unique characteris- 
tics and objectives of their project. Before presenting the activities and 
instruments, a rationale will be provided for using alternative ap- 
proaches to assess learners' ongoing progress. 

Characteristics of Alternative Approaches 
to Assessing Ongoing Progress 

Conventional assessments often use pre- and posttests to com- 
pare how much learners know before and after an instructional unit or 
employ standardized measures that may not reflect the form and 
content of the program. Alternative assessments can go one step 
further. Rather than focusing only on knowledge and skills, they seek 
to show changes in learners' attitudes, social interactions, and commu- 
nication patterns that may be attributed to their participation in the 
program. In contrast to many standardized assessments, alternative 
approaches can link instructional content with assessment. In addition, 
alternative assessments are often participatory, involving learners as 
partners in the assessment process. 

Linking Curriculum with Assessment 

Alternative assessments can focus on the way learners use 
language and literacy as opposed to how much they know about 
grammar, phonics, or spelling. For example, a traditional test might 
assess whether students know irregular verb tenses, can find words 
that rhyme with "cat," label the parts of the body, or answer compre- 
hension questions about a paragraph. An alternative assessment, on 
ihe other hand, might seek to examine how well a learner can explain a 
problem, read a report card, or write a note to a neighbor. Alternative 
assessments also can be used to report on the role that literacy plays in 
learners" lives and to document the life changes that take place as 
learners become involved in family literacy. 

Alternative assessments arc particularly useful with a language 
and literacy curriculum that emphasizes meaning over form and 
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IHjP During the early stages 
of a project, it is important for 
learners to have opportunities 
to discuss the role that 
literacy plays in their lives. 



communicaiion over structure. Such assessments are also effective for 
learner-centered approaches to teaching English-as-a-second-ian- 
guage (ESL) literacy. One recent research study on successful pro- 
grams and promising practices defined a learner-centered approach to 
ESL literacy as "supporting language-minority aduks with little 
English and litile formal education in their efforts to understand and 
use English in its many forms (oral and written, including prose, 
document, and quantitative literacy), in a variety of contexts (family, 
community, school, work), so that they can reach their fullest poten- 
tial and achieve their own goals (personal^ professional, academic)" 
(Wrigley & Guth, J 992, p. 7). 

Because a learner-centered curriculum tries to be responsive to 
the participants' changing needs and goals, learner-centered assess- 
ment also needs to be dynamic and flexible. Such assessment should 
be capable of capturing incidental learning and unforeseen problems 
and of demonstrating the extent to which program objectives have 
been met. A collaborative effort among teaching staff, learners, and 
those evaluating the project helps to ensure effective planning and 
implementation of alternative assessment (see Chapter II for a detailed 
description of the planning process). 

The next section is the first of six sets of instructional activities 
that illustrate a variety of alternative approaches to assessing ongoing 
progress. Linking instructional activities with assessment provides a 
multifaceted view of the learners* involvement and progress in a 
project. 

Helping Learners and Staff Get Acquainted 

During the early stages of a project, it is important for learners to 
have opportunities to discuss the role that literacy plays in their lives. 
Through a series of exploratory activities, learners may identify their 
goals for literacy, describe the ways they use literacy at home and in 
the community, explain their hopes and dreams for their children, and 
talk about their interests in acquiring literacy in their native language 
and their goals for learning English, This section presents activities 
and assessment procedures that can be used at the beginning of the 
instructional program. These activities also provide opportunities for 
teachers and learners to get to know each other during the initial 
stages of the project. In addition, they present a chance to discuss how 
the program can meet learners' needs and help families move toward 
their goals. (Chapter III provides initial assessment strategies for 
obtaining information for planning and developing instructional 
objectives and curriculum content) 

Surveys and questionnaiies can be used just before classes begin 
to assess the learners* needs. After instruction is underway, additional 
information can be gathered effectively in group sessions in which 
learners discuss the roles that English and literacy play in their daily 
lives. These sessions may result in literacy inventories that reflect 
when, where, and for what purposes learners engage in certain types 
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of reading and writing; for example, is reading religious material more 
important to them than reading the newspaper? Activities thai invite 
learners to talk about how they use literacy and what they want to be 
able to do with literacy should occur at regular intervals throughout 
the project- Such activities will help learners express their goals more 
clearly and identify ways in which literacy can enrich their lives. To 
develop useful learner-centered assessments, staff members need to 
gain ai l ! tiaintain the trust of the learners so that they feel free to share 
* Ur exper, nces, ideas, and aspirations with their peers. 

Smull-fc «ip and pair activities are especially helpful in allowing 
staff to ( ! ) .low learners interact with each other; (2) observe the 
learning slrat ,ies they use; (3) gain insights into the learners' com- 
Tiur.ication -ills and literacy practices (in English and in their native 
language)- .«id (4) recognize the skills and abilities that learners bring 
class. 

a.d v>ai uore about the learners and help them get to know 
^nc an; sta»"\ tubers can use ( I ) grids in which participants 
msi.i* " fr ..nation eboiit a classmale with the person's name: for 
example, "Who has two children and comes from Vietnam?"; (2) 
maps and pictures from magazines that allow learners to talk about 
their native countries; (3) photographs brought from home that enable 
learners to talk about their families; and (4) language experience 
stories in which learners tell about a significant event such as, "My 
first day in the United States." 

Learners also can get acquainted by interviewing each other. 
Gillespie (1990) suggests that teachers cut pictures in half; ask learn- 
ers to choose half a picture out of a box; find the person who has the 
other half; and interview that person. Interview topics could include 
questions about the learners: 

1. Interests, likes, and dislikes. 

2. Background knowledge, skills, and occupation. 

3. Reasons for enrolling in the projeei. 

4. Family life and pareal-child activities. 

5. Experiences with schooling in their native country and the United States. 

Follow-up activities can address such questions as, "When and 
where do you use English and why?"; "When do you use your native 
language?"; "What are your educational goals for yourself and for 
your children?". Learners also can be asked about the role of literacy 
brokers in the communis "ndividuals who act as cultural guides to 
U.S. society and help newcomers negotiate paths through the maze of 
U,S. bureaucracies and other obstacles (Weinstein-Shr, 1990). Of 
particular interest for family literacy might be the role that children 
play in mediating language for their parents and grandparents. Staff 
can use such investigations to create in-class discussions about cross- 
cultural differences in communication* schooling, or home health 
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care. These discussions could then lead to instructional activities that 
provide opportunities lor sharing feelings and ideas and for acquiring 
other knowledge and information. 

Assessment Strategies 

Activities in which participants interact, learn together, and teach 
each other provide staff with opportunities to assess what learners 
know, what role literacy plays in their lives, when and where they use 
English and their native language, and what educational values they 
hold. Collaboration among participants helps create a community of 
learners and allows staff to assess learners along various dimensions of 
literacy: 

1 . Levels of self-esteem and confidence in interacting with others. 

2. Ability to understand spoken and written English. 

3. Ability to express ideas orally and in writing. 

4. Patterns of native language use at home and in the community. 

5. Knowledge about cross -cultural differences in institutions, services, and 
life-skill areas. 

6. Levels of independence and degree of transfer of skills and knowledge 
beyond the classroom, including changes in behavior desired by 
participants. 

Staff members who are uncertain about which of these dimen- 
sions to focus on initially can design an observation measure for each 
learner that captures information in each of the areas until categories 
worth pursuing begin to emerge. Or staff can create a simplified 
version that would focus on fewer areas. Example 1 (see page 79) is an 
observation measure that can be used to obtain basic information about 
the learners. The information obtained can be used for the following: 

1 . To generate baseline data for monitoring learners' progress over time. 

2. To provide information for staff members to share regarding individual 
learners and to detect common patterns in the learners' responses to 
classroom activities. 

3. To create classroom activities that address learners' needs. 

Staff members can use questions such as the following to guide 
their discussion and to develop descriptors for various levels of 
progress; 

1 . What differences do we see bei*een advanced and lower level learners in 
reading, writing, speaking, and listening abilities? 

2. What evidence of coping skills do we observe? What do we know about 
the strategies that learners use to solve problems and communicate in 
English even though they have only minimal English and are just barely 
1 iterate? 

3. How much do individual learners already know about laws, regulations, 
and institutions that affect their community (for example, schools, health 
care providers, and social service agencies)? 
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4. What evidence do we have thai learners are applying in their personal 
lives what we are teaching in the project? 

5. What are the characteristics displayed by independent learners? How are 
such learners different tram those who are overly dependent on their peers 
or the teacher? 

By developing descriptors of learners' behavior at various stages 
of development, staff are able to identify changes in learners' perfor- 
mance over the course of the project. Staff members can continue to 
add meaningful information to learner profiles as the project is imple- 
mented. When assessment and instructional activities work in tandem, 
learners' needs and iheir progress are illuminated, enabling staff to 
gain clearer insights into how the project can be tailored more appro- 
priately to the learners. 

Figure 1 contains comments that staff members might write 
about learners in the learner profile (see Example I . page 79). 

Linking Project Activities with Family Life 

This section shows examples of how instructional activities, goal 
setting, and assessment can be linked with family life. First, learners 
work together to create some kind of joint project. Next, parents talk 
about the kinds of activities they do with their children at home and 
discuss how these activities can support children in their own learn- 
ing. Parents then decide what they would like to do with their children 
during the next class sessions. Finally, parents talk aboul the time they 
spent with their children and what their experiences have meant 
to them. 

What to Do 

Initial program activities need to encourage participants to learn 
from and leach each other by observing and doing. For example, staff 
members, parents, and children can work and play together to create 
collages, holiday greetings, announcements, or birthday cards. Staff 
might start by showing some interesting examples and by demonstrat- 
ing a particular technique for creating the materials. While they work 
with paper and scissors, participants learn the English names for 
colors, shapes, and actions such as cutting, folding, and tearing. 

In subsequent activities, participants share information, describe 
their materials, or teach each other a particular technique. Similar 
activities can include making paper airplanes, origami (paper folding), 
or party decorations. (For an example of a lesson on making and 
playing traditional instruments, see Quintero & Hueita-Macfas. 
1990.) Next, parents can discuss some of the activities they do with 
their children at home and talk about the kinds of reading and writing 
they do in their native language or in English. 
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Examples of Comments for 
the Learner Prof lie/Observation Card 

Communication skills 
Listening comprehension 

Felicia still looks a bit overwhelmed and confused when we discuss 
things or when I give directions for a particular task. She seems to 
understand more when I talk to her directly, especially when I ask her 
about her family. She does not seem to understand very well when we 
venture into topics that go beyond her family, her home, or the town 
that she came from (Guadalajara). 

Reading 

Felicia seems eager to read. Whenever I hand ouMhe language 
experience stories that the class has written, she immediately gets 
started. However, she seems to get stuck easily whenever she 
encounters an unfamiliar word or a difficult phrase. She is able to 
answer simple comprehension questions about a story we have 
talked about in class but has difficulty with new readings. She seems 
quite ready to link her own experience to other people's stories and 
her responses to the story focus on recounting the facts presented. I 
am hoping she will talk about herself more and see the connections 
between her own life and the experiences of other students in the 
class. Felicia picks up magazines during our free reading periods and 
leafs through them but she does not seem engaged in any of the 
texts. 

Coping strategies and problem solving 

Although Vuth scored fairly low on our standardized placement test, 
he seems to cope pretty well in class. He is able to figure out what to 
do when I assign a task by watching other students or asking an 
occasional question. He is willing to put pen to paper when I ask the 
class to write (although his written English is very difficult to 
understand), and he eagerly looks through the books and magazines 
we have in the classroom. Although he has a friendly smile, his tone is 
still rough at times, especially when he asks for something. (Teacher, 
give me book 11 is a typical request.) 

Content knowledge and understanding 
of institutions, services, and life skills 

Marta seems interested in how the community works, but she has 
very little knowledge of the services that are available to her (she did 
not know she could get free prenatal care at the clinic). A friend 
brought her to class, and she has only been in California for two 
months. She does not know the name of the school her children 
attend or the names of their teachers. She does know that 911 is for 
emergencies but thought that all she has to do is dial and the police 
will come right away to the right address (she was surprised to hear 
that she would have to explain the nature of the problem). She did not 
know that she could ask the operator for help in Spanish. 



Chapter IV ^ 



initial program activities 
need to encourage participants 
to learn from and teach each 
other by observing and doing. 



Activities that involve 
parents and children working 
together offer staff a starting 
point for discussing the social 
interaction among family 
members and for 
understanding the literacy 
practices used in the home. 



Assessment Strategies 

Activities that involve parents and children working together 
offer staff a starting point for discussing the social interaction among 
family members and for understanding the literacy practices used in 
the home. To leam more about the participants' families, staff can 
develop questionnaires in which parents, working individually, 
in pairs, or in groups, list the activities they do at home with 
their children. 

Example 2 (see page 80) is an illustration of a questionnaire for 
generating a list of activities that parents and children do together. The 
list generated by the learners can become baseline data for document- 
ing changes over time in parent-child interactions (see Example 3, 
page 8 1 ). The assessments shown in Examples 2 and 3 could be 
conducted in the learners' native language or in English. 

in ensuing activities, staff members can provide information to 
parents about learning strategies used by children and present ex- 
amples of how parents can support their children's learning at home. 
For example, parents can give children opportunities to observe others, 
learn new things with siblings, ask questions, or discuss cause and 
effect relationships. Using a questionnaire such as the one shown in 
Example 4 (see page 82), parents can discuss activities they consider 
useful for helping their children. 

Next, parents can identify the kinds of activities they plan o 
undertake with their children and the frequency with which they 
might conduct each activity (see Example 5, page 83). These activities 
could become part of an individualized plan that helps learners 
document progress toward meeting their own goals and the objectives 
of the project. Parents could use a questionnaire such as Example 6 
(see page 84) for reporting the activities they try with their children 
and for determining to what extent they have met the goals in 
their plan. 

f xamples 4-6 can be adapted to meet the specific purposes of a 
family liteiacy project. For example, the complexity of the English 
voeabulary and structures used in the instruments could be tailored to 
the learners' ability. In addition, the instruments could be translated 
and administered in the learners* native language. Some projects also 
might have staff members or learners use the instruments for indi- 
vidual interviews in English or the native language. 

Reading to Do 

Beginning learners need opportunities to deal with different 
kinds of texts and cope with a variety of forms of English. Storytelling 
and language experience activities show how sentences arc connected. 
To focus on one word or sentence at a time, learners can read signs, 
symbols, and instructions that appear in their environment everyday. 

The activities in this section are designed to help participants 
develop their receptive skills of listening and reading. The focus of the 
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activities is on introducing beginning learners lo reading through 
environmental print (signs and symbols in the community) and 
imperative print (messages that tell citizens what to do and not to do). 
An observation measure is presented for assessing listening and 
reading abilities. 

What to Do 

Field trips ean be used to give adult learners opportunities to 
operate a variety of machines such as those for making photocopies or 
obtaining soft drinks and change. Participants also can learn how to use 
a public telephone or operate an automatic teller machine. 

Working in pairs or small groups, learners discuss the instruc- 
tions for the machines and practice what is required to operate them. 
Learners might use a Polaroid camera for taking pictures of instruc- 
tions they find hard to understand. During a field trip, they also can 
write examples of imperative print, such as Exit, Stop, and No Turn 
on Red. Staff members could select several sets of the instructions for 
additional practice in class. 

During classroom activities, learners can identify the machines 
they found particularly useful, interesting, or difficult to operate. They 
might decide which instructions they would like to study further in 
literacy activities such as reading simple imperatives, matching 
instructions with appropriate machines, or putting instructions in 
sequence. For example, the sequence of instructions for using a public 
telephone would be ( 1 ) lift receiver; (2) listen for the dial tone; (3) 
deposit 20 cents; and (4) dial the number. 

Follow up. Learners can develop language abilities related to 
operating other machines such as an audiotape recorder, camera, 
videocassette recorder, three-hole punch, subway ticket machine, or 
computer. As learners become more proficient, they might work with 
longer and more complex instructions for such tasks as assembling 
simple toys, making crafts, following recipes, or completing job 
application forms. 

Assessment Strategies 

As learners try to figure out how things work by reading or 
listening to instructions, staff members can observe their reactions and 
assess the learners* reading and listening skills. Similar activities 
should be repeated throughout the instructional cycle to determine 
whether reading and listening are becoming easier and to compare 
learners* skills at the beginning of the course with their abilities 
toward the end. Instructions can be given orally to test listening 
comprehension or in writing to assess reading skills. In both cases, the 
instructions used for assessment should be somewhat familiar lo 
learners so they do not become overwhelmed or discouraged by the 
task. Example 7 (see page 85) is an illustration of an observation 
measure for assessing listening and reading abilities. 
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To develop a consensus on the meanings of the descriptors used 
( With ease, With some support, etc.) and to establish the reliability 
of the observation measure, staff members should use the instrument 
with adult learners at similar levels of proficiency under a variety of 
circumstances. With practice, learners can develop their own under- 
standing of the meanings of the descriptors and begin to evaluate their 
own performance. At more advanced stages, they can evaluate their 
peers and give constructive feedback about their performance. 

Writing to Do 

Writing instructions for others to follow provides learners with 
the opportunity to communicate ideas to an audience. Because instruc- 
tions require clear language and a logical sequence, they represent an 
effective way for learners to write down what they know and to ensure 
that their readers understand. 

This section presents cooperative activities in which participants 
teach and learn from each other by talking about a particular skill or 
ability they possess. The activities work well in a multilevel class, 
because they give less proficient learners a chance to develop knowl- 
edge by observing and responding nonverbally while more advanced 
learners enjoy the special challenge of teaching others. An observa- 
tion measure is presented for assessing learners' speaking and writing 
abilities. 

What to Do 

In cooperative activities, participants realize that each member 
of the group can be a teacher and a learner. To introduce the activities, 
staff members can ask parents what they teach their children and what 
their children teach them. It may be interesting for adult learners to 
talk about what they learned from their parents. Learners might also 
discuss how they help their neighbors and friends. These activities 
allow staff members to identify skills participants have, such as 
repairing a flat tire, preparing a home remedy for a sick child, fixing a 
broken window, getting rid of stains, or braiding long hair. 

During these activities, staff members can validate knowledge 
learners possess such as knowing how to find a bilingual doctor, 
where to get free legal advice, how to talk to a social worker, and 
where to find a video store that stocks films in the learners' native 
language. Staff members might make a list of skills represented in the 
group and ask learners to decide which ones they would like to 
develop further. Staff can also discuss the possibility of bringing in 
outside speakers from legal aid and public health agencies, the Red 
Cross, or the community relations unit of the police department. 
Presenters can talk about topics such as fighting an eviction notice, 
practicing safe sex. giving blood, or protecting a home against 
burglaries. 



Assessing Success 



81 



o 

ERIC 



BEST COPY AVAILABLE 



Staff members can ask participants to choose a particular task 
that interests them such as making tamales or a fruit salad, sorting 
coupons to use at the grocery store, demonstrating first aid, playing 
the kazoo, making popcorn, putting up a bulletin board* or taking 
Polaroid pictures. Group members might work together to write 
instructions for others to follow. After discussing and demonstrating 
the instructions in small groups, individual learners can demonstrate 
the task to the class using both verbal and nonverbal cues to illustrate 
their points. To evaluate the learners' speaking and wiring abilities 
related to giving instructions, staff members can use an observation 
measure similar to the one shown in Example 8 (see page 86). 

Assessment Strategies 

The observation measure shown in Example 8 demonstrates how 
an instrument can be developed for specific purposes in a pruject. In 
this case, the instrument was designed to measure learners' progress in 
improving their speaking and writing abilities related to givir ; 
instructions for particular tasks. 

Another assessment strategy would be to put each learner's 
written instructions in a portfolio for comparison with future writing 
samples. Copies of the instructions could be given to other learners to 
try to follow. Staff might use selected sets of instructions for subse- 
quent literacy activities such as fill-in-the-blank or cloze exercises, 
completing sentences that require longer phrases, unscrambling words 
in sentences, sequencing instructions, and matching instructional steps 
with pictures. 

For intermediate learners, the types of activities described here 
can be extended to teach functional literacy skills such as filling out 
school registration forms, understanding immunization records, and 
telephoning the school for information. (For other examples of 
literacy activities for intermediate students, see Holt's Parenting 
Curriculum for Language Minority Parents: Teacher's Activities 
Guide. 1988.) 

Staff members also can design writing activities for the whole 
class. For example, in preparation for a party or a holiday, learners 
might work together to prepare a meal, provide entertainment, explain 
the significance of the day, write invitations, and make decorations for 
the room. The activities could then be written up, either as work plans, 
narratives, or items for a class newsletter- The written work can 
become the basis for further language and literacy development. 

Building Self-Esteem and Taking Literacy 
Beyond the Classroom Walls 

The alternative assessment instruments discussed thus far focus 
on language use; that is, they seek to assess how well learners ean 
perform specific language-re laled tasks. Although these instruments 
can document changes in language abilities, they do not provide 
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information on affective factors such as self-confidence, self-esteem, 
or motivation. Nor do the instruments provide data that would indicate 
how well learners work together or lo what extent they transfer 
knowledge gained in the classroom to their lives beyond ihe school. 

Affective factors deserve the attention of staff members because 
of the importance of self-confidence and self-esteem in influencing not 
only the learners' language development but also their general level of 
success in adapting to the cultural challenges they face at home, in the 
community, and at work. Research has shown that learners who 
interact with and use ethers as resources make significantly greater 
gains in learning than do students who work alone (Kagan, 1 986: 
McGroarty, 1993). Similarly, studies in cognition demonstrate that 
language and literacy development is increased if learners can link 
classroom knowledge to real life contexts (Anderson, 1988; Anderson 
& Pearson, 1988; Smith, 1975). 

What to Do 

Project staff can use a variety of activities to help learners de- 
velop self-esteem and to improve their interaction with others. Activi- 
ties should enable learners to demonstrate the knowledge, insights, and 
abilities they have developed in responding to real life challenges. The 
following are activities that staff might try in their project, or they 
may want to design their own. 

1 . Case studies in which learners are presented with a familiar problem and 
asked to brainstorm possible solutions. For example, a case might involve 
a daughter who wants to play on the volleyball team, but the mother, a 
single parent, is worried about her daughter's being away from home for 
practice and out -of -town games. 

2. Situations that illustrate a significant economic, social, or cultural issue, 
such as lack of affordable housing, street crime, and racial, ethnic, and 
language-related discrimination. The situations, sometimes called codes, 
can be presented through short readings, dialogues, and pictures. They 
can provide starting points for the learners to discuss their concerns and to 
brainstorm solutions. Codes arc based on problem-posing pedagogy 
developed by Paulo Freire. (See Wailerstein, 1983 for similar activities.) 

3. Simulations and role plays in which learners play themselves in situatiuns 
such as performing a cultural dance, participating in a parent-teacher 
conference, or dealing with uncooperative landlords, 

4. Projects such as developing a class biography, producing a resource guide 
for newcomers to the community, organizing activities thai help their 
children learn their native language, or producing dictionaries and 
pictionaries in English and the learners' native language. 

5. Lists and charts of individual accomplishments that are posted on 
classroom walls. Learners use their own word to describe achievements 
related to language, literacy, and learning that they arc proud of. 

6. Language experience stories in which learners tell of the struggles in their 
live* and celebrate their successes in becoming bil iterate and bitultural. 
(For examples of I earner* generated stories, sec Wein&lcm-Shr, 1992.) 
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Assessment Strategies 

The observation measure shown in Example 9 (see pages 87-88) 
can be used to ( 1 ) record changes in learners' confidence; (2) docu- 
ment levels of participation in pair or group activities; and (3) show 
transfer of skills to actual life experiences. By recording information 
about the three affective areas in one chart, staff members can analyze 
relationships among the domains. For example, the data might indi- 
cate that as learners become more confident, they participate more 
freely and transfer the knowledge gained in class more readily to real 
life. Results of the measure might also illustrate that levels of self- 
confidence are related to levels of participation, as in the case of learn- 
ers who collaborate more freely as they become more self-confident. 
When interpreting the results, staff members need to be aware that 
factors outside the project also may influence the learners* level of 
self-confidence. 

Developing Literacy Through Language 
Experience 

One of the most challenging tasks faced by literacy project staff 
is introducing nonreadcrs to literacy. Adults who are developing 
literacy in English as well as in their own language need a great deal 
of lime and special support so they can comprehend printed materials 
and express their views in writing. This section gives examples of 
activities designed to facilitate learners' literacy development. Instru- 
ments and procedures are presented for holistic assessment of writing 
samples. 

What to Do 

Many useful strategies are available for helping learners acquire 
literacy. An effective approach is to begin by helping learners read 
words Ihey already know, such as their own names, the names of their 
children, iraffic signs, stcre marquees (e.g.. K-Marl), and product 
logos (e.g., Coca-Cola). Activities based on environmental print can 
provide an excellent starting point for developing literacy. 

For leachers who believe that instruction in phonics is an impor- 
tant part of introducing learners to literacy, phonics activiliescan be 
integrated easily into authentic reading and writing activities. For 
example, to help her students move from oral language to literacy. 
Dean (1992) suggesls using popular songs as a starting point for 
literacy activities. She uses the Mexican song "La Samba" to help 
learners associate sounds with print and to demonstrate that many 
syllables in Spanish consist of similar letter combinations: for ex- 
ample, la; bam-ba: ma-ri-ne-ro. 

Another promising way to introduce reading is the Language 
Experience Approach (LEA). LEA represents both a philosophy of 
leaching and learning and a method for connecting oral language lo 
literacy. Il involves a staff member or another learner transcribing a 
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slory lhal a learner or a group of learners has created based on an 
individual or shared experience. The slories are then read aloud and 
shared with group members. After the learners become comfortable 
with the content and language, the stories can provide the basis Tor 
group discussions, additional reading and writing activities, or in 
some cases, language practice. Because the language used in these 
stories comes from the learners, the vocabulary is familiar, the stories 
are reflective of the learners* experience, and the sentence structures 
are comprehensible. Some literacy projects evaluate the learners' 
progress in acquiring language by assessing the LEA stories they 
generate. (Sec also Taylor. 1993.) 

Assessment Strategies 

Language experience slories can be used as starting points for 
assessing reading and writing for learners at lower levels of profi- 
ciency. Other assessment strategies may include interviews that ask 
readers to talk about what they like to read and how they derive 
meaning from print. Also helpful are surveys in which learners are 
asked to list the kinds of reading and writing they do in the course of 
their daily lives, for example, making grocery lists, checking televi- 
sion viewing schedules, reading the newspaper, observing traffic 
signs, and scanning school notices. 

As learners new to literacy make progress in ihe project, assess- 
ment may focus on showing how the range of literacy materials 
increases as learners become more efficient readers and writers. At the 
end of an instructional cycle, even beginners may be able lo read signs 
and ads as well as simple notes, cards, stories, and poems. By observ- 
ing reading behavior and asking learners to describe the strategies 
they use for deriving meaning, staff can document increases in the 
learners' range of I iteracy practices. 

Learners also will increase their range of writing abilities. 
Slowly, they will write letters with more ease and require less effort to 
get words down on paper. For example, a learner may move from 
writing her name in big letters on a name card to signing a birthday 
card. As learners become more comfortable expressing their ideas, 
their writing may show a strong voice that evokes the reader's emo- 
tion. A mother may write "My children far away. I very sad. I wish we 
together." Although the structure of these sentences is not grammati- 
cally correct, the piece has a focus, shows logical coherence, and 
clearly communicates the loss the writer feels. 

Two scoring methods can be employed for scoring learners" 
work: holistic and primary trait. Holistic scoring provides an overall 
view of the learners' current writing ability. Scales may range from 
four point up to ten point, For example, a rating of four on a four-point 
scale could mean a more than aaequaie response in terms of organiza- 
tion, clarity, content, and mechanics; three, an adequate response; 
two. less than adequate; and one, very inadequate. 
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As learners new to 
literacy make progress in the 
project, assessment may 
focus on showing how the 
range of literacy materials 
increases as learners become 
more efficient readers and 
writers. 



Two scoring methods 
can be employed for scoring 
learners' work: holistic and 
primary trait. 



Primary trait scoring analyzes specific dimensions, or traits, such 
as voice, sentence structure, conventions (e.g M capitalization and 
spelling), and grammar. For any one writing assignment, several traits 
or just a small number can be measured. Each trait can be scored on a 
four- or six-point range. The learners' score can be based on indi- 
vidual traits or on the sum of the scores for all of the traits. Example 
10 (see page 89) is an example of a primary trait scoring rubric. 

Regardless of whether holistic or primary trait scoring is used, 
staff members need to be trained in the scoring procedure. Training 
should involve staff members working together in a group to score 
some wi lting samples and then score samples individually. Individual 
staff members' scores should agree before they start scoring their own 
learners' work. Periodic checks should be made to ensure that staff 
members continue to agree on what constitutes a two or a three, for 
example, and on what scores should be given to specific writing 
samples. 

Example 10 is an assessment instrument designed to evaluate 
writing samples according to the dimensions of ( 1 ) authenticity, voice, 
and engagement of the reader; (2) focus, organization, and develop- 
ment; and (3) sentence mechanics and language. The example was 
adapted from an instrument that appears in Soifer el at. (1990). 
Chapter V presents strategies for collecting, analyzing, and reporting 
data collected with Example 10. 

Beeause progress in writing is often difficult to observe from t!,iy 
to day, staff may want to keep a collection of learners' work in indi- 
vidual portfolios. Writing samples for the portfolio may be chosen and 
evaluated at key points during the instructional eycle. At specified 
intervals (for example, once every three weeks), staff members might 
identify several examples of learners* writL* that were produced 
during that time period. Staff could share their impressions as they 
review the writing samples. Staff members might invite learners to 
one-on-one conferences in which learners select whal they regard as 
their best or favorite piece of w riting and talk about the progress they 
have made in the project. 

Assessing the progress of new readers and writers. A ssessi ng 
the progress of beginning readers and writers can be a tricky undertak- 
ing. Three things need to be kept in mind: First, because reading and 
writing involve thinking processes that are not always observable, 
teachers need to be open to surprises and develop strategies for finding 
out how learners are thinking as they read and write. Second, reading 
and writing development are not linear processes; rather, literacy 
develops in fits and spurts. As a result, a learner might appear to be 
fairly high on a reading and writing scale one week and lower the next. 
Third, learning how to read and write is « comple* process that 
requires special skills and strategies — for example, being able to 
connect meaning to print and understanding thut writing represents 
speech. 
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Because progress in 
writing is often difficult to 
observe from day to day, staff 
may want to keep a collection 
of learners' work in individual 
portfolios. 



Because reading and 
writing involve thinking 
processes that are not always 
observable, teachers need to 
be open to surprises and 
develop strategies for finding 
out how learners are thinking 
as they read and write. 



In assessing the 
progress of beginning literacy 
students, it is important tor 
staff to be aware of the 
cyclical and recursive nature 
of literacy development. 



Literacy also involves subskills such as understanding the 
relationship between letters and sounds (phonics) and forming ihe 
letters of the alphabet. Another important literacy skill is sometimes 
referred to as making meaning — that is, the ability to make sense of 
written words or to express thoughts in writing. The check list shown 
in Example 1 1 (see pages 90-9 1 ) is designed to assess the meaning 
making strategies associated with literacy. 

In assessing the progress of beginning literacy students, it is 
important for staff to be aware of the cyclical and recursive nature of 
literacy development. Staff members should not be surprised if a new 
reader understands quite a bit of one story but becomes confused or 
stymied by another. Similarly, a writer may feel inspired by a particu- 
lar topic, yet have nothing to say on a different theme. 

Understanding some of the factors that help or hinder compre- 
hension and writing proficiency helps staff assess literacy progress. 
The learners' background knowledge, goals, and interests influence 
how well they are able to deal with a particular text. Thus, staff need to 
observe and listen to the learners to get a sense of the kinds of topics 
they may want to read and write about. Similarly, staff should provide 
a wide range of literacy activities (for example, listening to songs, 
labeling photographs, illustrating a story, or reading about others in the 
class) so they can respond effectively to the participants' different 
learning strategies. Staff also should look at many different examples 
of what learners are able to read and write to get a complete picture of 
the learners' literacy development. 

The check list shown in Example 1 1 provides descriptions of 
some of the strengths that learners exhibit as they go about making 
meaning in reading and writing. It illustrates some of the learners' 
characteristics as they move from being relatively unfamiliar with 
literacy to becoming independent readers and writers. Not all learners 
will exhibit all the descriptors in the instrument. As staff members 
read and write with their learners, they should check the descriptors 
that characterize a particular learner. They can then compare the 
check lists of several learners to determine how the learners as a group 
are developing from emerging to independent literacy. Staff members 
also can compare the check lists of several learners and summarize the 
data ai the end of an instructional cycle. 

Summary 

Adult learners have their own reasons for coming to class, only 
some of which have to do with acquiring English and developing 
literacy skills. Effective assessment approaches need to take into 
account the learners 1 needs and goals and develop ways of showing 
Ihem that they are making progress. 

Alternative assessment approaches enable slaff to create project* 
based instruments and involve learners in developing the curriculum 
and designing the evaluation. The instruments may include observa- 
tion measures, protocols for interviews and focus groups, learner 
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profiles, and performance samples. Staff will want to use and field test 
these instruments to measure progress reliably overtime. The data 
gathered with the instruments can then be aggregated into summary 
formats and, where appropriate, quantified so that the information can 
be used to answer a variety of questions about the project- 
Staff members who use alternative assessments that are project 
based and learner focused have access to more valid and useful 
information than that which can be obtained from most standardized 
tests. To improve the use of alternative approaches to assessment and 
evaluation, project staff face the challenge of defining their vision of 
success and developing instruments that capture the changes that take 
place as language-minority adults and their children become involved 
in family literacy. 
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Example© 



Learner Profile/Observation Card 



Type of assessment: Observation measure. 




Purpose: To record information about the knowledge, skills, and strategies that learners possess and 


the level of coufidence they display in interacting with each other and to collect anecdotal evidence 


that shows the extent to which participants are using the information gained in class in their lives 


outside the classroom. 




Method: Open-ended charts allow teachers to take notes on what learners do and say. 


Learner Profile/Observation Card 


Date 


Teacher 


Unit 


Class 


Learner's name 




• Communication skills 




listening and reading 




speaking and writing 




• Coping strategies and problem solving 




linguistic 




nonverbal/social 




• Content knowledge 




institutions 




services 




life skills 




• Levels of confidence 




self 




onc-on-one 




small group 




large group 
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Example© 



Questionnaire 



Type of assessment: Self-iissessmeit; questionnaire. 

Purpose: To idenlify literacy practices in the home and record parent-child interactions. Information 
provides baseline data for documenting changes over time. 

Method: A list of activities is generated as part of whole-group or small-group discussions or after 
learners interview each other. If learners are not proficient in EngliJi, the activity can take place 
in their native language. In cases where the learners are not literate, the teacher or a literate 
partner can record responses. 

Prompt to learners: Parents and children can do many things together. They go to the park on 
Sunday, go fishing, cook food, clean the house, go hiking, watch TV, work in the garden, or look 
at magazines. In many families, parents help their children with homework or check their assign- 
ments. What do you do together with members of your family? 
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Example 0 

Self-Assessment Instrument 



Type of assessment: Self-assessment of literacy activities by frequency count. 

Purpose: To identify family activities that learners engage in at home. The instrument can be used for 
generating baseline data and documenting changes during the implementation of the project. 

Prompt to learners: You do many things together as a family. Tell us how often you do these activities. 
(Be sure to list any activities you do that are not on the list.) 

Almost Once or Several Once or 

Family activity every day twice a week times a month twice a month Seldom 

Go to the park 

Watch TV together 

Cook or clean house together 

Fix things that are broken 

Go fishing 

Do homework together 
Look at a book or magazine 
Play games 
Work in the garden 
Go hiking 

Attend cultural events 
Other activities 
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Example O 



Questionnaire 

Type of assessment; Seif-assessment; open-ended quest ion nahc. 

Purpose: To record home literacy events and activities that parents regard as essential and to gain 
insights into educational values and opinions about learning. 

Method: As part of whole- or small-group discussions or through pair activities, learners discuss their 
views of how children learn and how parents can help their children learn. 

In my opinion, these activities help children learn: 

Talking about (for example, field trips and birthday parties) 

Teaching children to (e.g.. ride a bicycle and bake cookies) 

Helping children with (e.g . writing a letter and doing math problems) 

Asking children questions about (e.g.. their friends, favorite teacher, and personal problems) 

Telling children that (e.g., everyone makes mistakes and doing homework is important) 
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Example 0 

Self-Assessment Instrument 



Type of assessment: Self-assessment; frequency count. 

Purpose: To set personal goals and develop an individualized plan for documenting progress. 

Prompt to learners: Based on the home literacy activities identified in the questionnaire (Example 4), list 
activities that you plan to undertake with your children and indicate how often you might do each. 



Almost Once or Several Once or 

Family activity everyday twice a week times a month twice- a month Seldom 

My children and I plan to . . . 



go to the library 
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Questionnaire 



Type of assessment: Self-assessment; open-ended questionnaire. 

Purpose: To record literacy events and learners" accomplishments related to literacy and to compare 
goals in the learners' individualized plans with actual activities. 

Method: Learners talk about the kinds of activities they did together with their children and discuss the 
literacy events that have taken place. Each learner or a literate partner records responses. At key 
points in the cycle the list is compared with the learner's individualized plan. The plan is then 
discussed anew and goals are renegotiated based on the learner's current perspectives. 



Learner's name 
Date 



7771a week, I helped my son with his math. 
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Example O 



Observation Measure for 
Listening and Reading 



Type of assessment: Observation measure. 
Purpose: To assess listening and reading skills. 

Method: After learners listen to or read instructions, staff members assess their performance and check 
the appropriate box in the chart. 

For example: After learning how to make instant chocolate pudding for a class party, learners are asked 
to follow oral or written instructions for making vanilla pudding (familiar instructions) and gelatin 
dessert (new instructions). 

Learners who can comprehend and read instructions without asking questions or looking terribly 
puzzled are rated "With ease." Those who occasionally ask to have a wor J provided, a sentence 
modeled, or seek help from peers are rated "With some support." Learners who need to consult a 
dictionary, ask peers for translation, or constantly observe others are checked "With great difficulty." 



With some With great Not at 
Receptee skills With ease support difficulty alt 

Listening Follows familiar instructions 

Follows new instructions 
Reading Reads familiar instructions 

Reads new instructions 



no 
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Example © 



Observation Measure for 
Speaking and Writing 



Type of assessment: Observation measure. 
Purpose: To assess speaking and writing skills. 

Method: As learners talk about or write down instructions, staff members assess their performance and 
check the appropriate box in the chart. 

For example: After learning how to make buttered toast, learners are asked to follow oral or written 
instructions for making toast with peanut butter and jelly (familiar instructions) and bagels and cream 
cheese (new instructions). 

Learners who can produce and write instructions without asking questions or looking terribly puzzled 
are rated "With ease." Those who occasionally ask to have a word provided, a sentence modeled, or 
seek help from peers are rated "With some support," Learners who need to consult a dictionary, ask 
peers for translation, or constantly observe others are checked "With great difficulty i 



- — r 



' | " 

With some With great 
Productive skills With ease support difficulty 




Not at 

ali 



Speaking 



Produces familiar spoken 
instructions that are 
clear and comprehensible 



Produces new spoken 
instructions that are clear 
and comprehensible 



Writing 



Produces familiar written 
instructions that are clear 
and comprehensible 



Produces new written 
instructions that are clear 
and comprehensible 
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Observation Measure for 
Nonlinguistic Domains 



Type of assessment: Observation measure. 

Purpose: To assess learners 1 performance in nonlinguistic domains related to learning 
and literacy development. The instrument can be used for generating baseline data 
to document changes during the implementation of the project. 

Method: Staff members write comments in the appropriate columns as they listen to 
learners talk about their abilities and work together in pairs or in small groups. 
Anecdotes about learners' lives outside the classroom are also used as sources of 
assessment data. 

The following are detailed descriptors for the domain Confidence that appears in 
the observation measure. Staff can develop their own descriptors based on the 
specific objectives of their project. Numerical values can be attached to the 
descriptors to facilitate data analysis (see Chapter V for strategies for analyzing 
data). 

Descriptors for Confidence 
Level 1 

Appears shy and unsure of own abilities; does not volunteer information; reluctant 
to answer if called on; seems overwhelmed by most new tasks; needs a lot of 
support. 

Level 2 

Acts confidently when handling familiar information and seems willing to take 
risks; freely talks about himself/herself and shares information with the class; needs 
additional challenges to grow and learn. 

Level 3 

Extremely confident; has a tendency to take over the class and sometimes show off, 
causing slight resentment in the other students; requires challenging tasks and 
needs to be in groups where he/she cannot dominate easily. 

Similar descriptors can be developed for the other two nonlinguistic domains in the 
measure, Participation and Transfer to real life. 
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Example 9: Observation Measure for Nonlinguistlc Domains (continued) 

I 



Date 


! ' 

Teacher 




Unit 


; Class 




Confidence 


i 

Participation 


Transfer to real life 


Descriptors: 


Descriptors: [ 


Descriptors: 


• Appears shy and unsure of 

UWI1 UUlHUCa 


• Appears overly dependent 

Ull UI1C Ul lwu iricnus 


• Does not yet use literacy 

Mil 11a UUlblUC lilt- Uclaa 


or 

• Acts confidently when on 
familiar ground and seems 
willing to take risks 

or 

• Extremely confident; has u 
tendency to take over the 
class 

i 


or 

• Uses the group as a 
resource and acts as a 
resource \ 

or 

• Likes to work indepen- 
dently most of the time; 
does not enjoy group work 

i 

-T-nr^nrr ~r ~n — r)prjim~i — m — rwr 


or 

• Practices at home 

or 

• Brings examples and 
questions from home to 
the class 


— — r- — ; : ■ ^ v ^ TC ^._^i. T . ... 1 
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Writing Assessment Instrument 



A. Authenlicity/voice/engagemeni of the reader 


, 4 = Expression strongly reflects the writer's emo- 
tional and/or intellectual involvemenl in the topic. 
Strongly engages the atlention oi the reader. 


Comments: 


, 3 = The writer is engaged in the topic and engages the 
| reader, 

r 2 = Uninteresting; not engaging; perfunctory. 

1 = Writing seems to be a mechanical exercise. 
Marked by cliches, hazy generalizations, mean- 
! ingless expressions. 


B . Focus/organization/development 


-i — — 

4 = Focuses on one main idea. Has clear beginning, 
middle, and end. Well organized and well 
developed through examples, 


Score: 
Comments: 


3 = Focused and organized but may have a flaw in 
coherence or incomplete elosure. Incomplete 
! development. Explanation is strongly implicit. 

2 = Lack of clear focus, organization, or develop- 
ment. Narrative, but no explanation. 

1 - Disorganized; underdeveloped; unconnected 
generalizations. 


C. Sentence mechanics/language 


4 = Few mechanical, usage, or sentence errors. 
Language used with fluency and variety. 


Comments: 


3 = Some minor mechanical usage or sentence errors. 
Language used competently to express ideas. 




2 = Enough usage errors to attract attention away 
j from the content Sentences understandable, but 
i unconventional. 

1 - Language and mechanical errors impair meaning. 




Reprinted by permission of the publisher from Soifer, Rena et al., The Complete Tkeory-ro-Prac'ice Handbook of 
Adult Literucy ©1990 by Teachers College, Columbia University. All rights reserved, (pp. 1 78-1 79) 
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Example® 



Checklist for Assessing Beginning 
Literacy and Emerging Literacy 



Learner's name: Date: 


Part I: Assessing Reading 


A. 


As literacy starts to emerge, the new reader: 




1- 


Is aware that print carries meaning but only in a very vague sense. 




2_ 
3. 


Can tell print from nonprini (e.g., decoration) but insists that she/he cannot read 
anything. 

Can tell the shape and form of English writing from nthcr types of writing but resists 
making a guess about the meaning of a word. 




4. 

5. 


Recognizes her/his own name and a few sight words but resists predicting what a new 
word might mean. 

Can interpret print that is part of the envirunment. such as reading a stop sign, 
recognizing a Coca-Cola can, pointing out the McDonald's logo, or interpreting a No 
Smoking sign, as long as a great deal uf suppurl is given. 




As literacy continues to develop, the new reader: 




I. 


Is able to read familiar sighs, labels, and logos as long as they appear in their natural 
form (e.g.. "rice' , printed on bag nf rice): is willing to experiment and venture some 
guesses. 




2 
3. 


Recognizes familiar words when written on paper, as long as sume context is pmvided 
(e.g., a picture of a bag of rice with "rice" written next to it). 

Can read short familiar texts, such as simple language experience stories, along with 
other group members. 




4. 

5, 


Employs reading strategies that allow her/him to move through familiar texts somewhat 
independently. 

Uses reading strategies that allow her/him to derive some meaning trom new texts, that 
is. is beginning to read independently. 


Part II: Assessing Writing 


A. 


As literacy emerges, the new writer; 




I. 


Is not yet aware of the functiuns and uses nf writing. 




2. 
3. 


Is aware that writing carries meaning but is not yet ready tu try to write. 
Copies letters and wurds but i* nut able tu read what she/he writes. 
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Example 11: Checklist for Assessing Beginning Literacy and Emerging Literacy (continued) 



4. Tries to write something, even if only a few scribbles; talks about what she has 
written; understands that writing is used to express ideas. 

5. Writes her own name independently, as well as some other key words; however, 
writing still seems to be a mechanical exercise. 

B. As lite, cy continues to develop, the new writer: 

1 . Writes or copies words and is interested in what the words inean; tries out some of 
her/his own ideas as long as some support is given; may copy a word and use it to 
label a picture or photograph. 

2. Writes a few words or sentences independently without copying; writing may be 
uneven and meaning not always clear. 

3. Experiments with writing on her/his own; may write a few words about herseif/ 
himself or may write about what she/he sees in a photograph. 

4. Experiments further with expressing ideas; writes a few words in dialogue journals 
or attempts to write parts of a language experience story; may need a great deal of 
support. 

5. Writes notes in dialogue journals and writes some parts of language experience 
stories independently; writing shows beginning elements of organization. 

6. Connects ideas and thoughts in writing; may write short description of a picture or 
an event; writes short description of self or of family; spelling and sentence 
structure still very uneven. 
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QSSESSING &UCCESS 



CHAPTER V 

COLLECTING, ANALYZING, 
AND REPORTING 
ALTERNATIVE 
ASSESSMENT 
RESULTS 



This chapter deals with collecting, analyzing, using, and reporting data 
obtained with four alternative approaches to assessment and evaluation: 
surveys,* performance samples, interviews, and observation measures. A 
variety of different analyses, techniques, and procedures are presented that 
can be used with each of the alternative approaches. However, the strategics 
presented for the four instruments also may be used with other types of alter- 
native assessments such as portfolios and learner investigations. This chap- 
ter illustrates how alternative assessment data can be analyzed, interpreted, 
and reported based on the characteristics of family literacy projects. 

The chapter is divided into four secticas, each of which addresses a different 
alternative approach. Each section deals with the four key aspects of manag- 
ing the data obtained from the alternative approaches: ( 1 ) collecting the data; 
(2) analyzing the data; (3) using the data; and (4) reporting the data and find- 

♦ See ihe Glossary an pages 137-138 of ihis publication for definitions of Mtcizrd key terms thai 
appear in this chapter. 




Sal Gelardi 
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One of the important 
advantages of alternative 
assessment approaches is that 
they offer staff the opportunity 
to involve learners in the 
assessment process. 



ings. Strategies are presented for a variety of stakeholders such as 
project staff members, learners, advisory groups, coordinating agen- « 
cies, and representatives of the funding agency. Additional informa- 
tion about alternative approaches to assessment and evaluation is 
available from such resources as Qualitative Evaluation Methods 
(Patton, i980) and A Practical Guide to Alternative Assessment 
(Herman, Aschbacher, & Winter, 1992). 

Data obtained through alternative assessment approaches are 
useful in the preplanning, planning, and evaluation phases of family 
literacy projects. Alternative assessment instruments are particularly *• 
useful in giving staff information necessary for (1) planning a pro- 
gram based on the needs of adult learners; (2) aligning curriculum . 
according to the results of learner assessments; (3) uionitoring learn- 
ers" progress toward attaining project objectives; and (4) involving 
learners in determining their progress toward mutually agreed goals. 
(See Chapters I and II for a discussion of objectives.) 

One of the important advantages of alternative assessment 
appreaches is that they offer staff the opportunity to involve learners 
in the assessment process. Learners may be involved in such tasks as 
identifying topics or skills that will be assessed, monitoring their own 
progress toward attaining project goals, or collaborating to determine 
their rating on a performance sample. As learners collaborate in the 
assessment process, they take more responsibility for their own 
learning and for the project in general. 

Surveys 

Surveys may be used to collect information for the preplanning, 
planning, and evaluation phases of the planning process model 
presented in Chapter IL In the preplanning phase, survey data are used 
to identify the needs of potential participants and to design objectives 
for the project. During the planning phase, survey data may be used to 
develop curriculum and instructional activities. This section addresses 
the use of survey results in the evaluation phase of the project. The 
information presented is based on the needs assessment survey 
presented in Chapter III, Example 1 (shown again on page 95). 

Co' ;cting Survey Data for Evaluation 

The responses to each question on the survey should be tabulated 
to determine the range of responses and the number of respondents 
who selected each response. For example, the first question on the 
survey asks the respondents to indicate the number of adults who live 
in their apartments. The range of responses for this question would 
consist of the smallest and largest number of adults living in one 
apartment. A frequency count would indicate how many respondents 
selected each response. Data should be tabulated similarly for each 
item in the survey. The frequency and range of responses should be 
tallied for each question according to the type of response choices (for 
example, yes/no, names of schools, or morning/afternoon). After the 
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ExampteQ 
Needs Assessment Survey 



! 

Nun* [ Telephone ( ) 

Addrev^ 

Nanw Language . _■ 

1 How nr.ny iiOuhs luc in>ivurapartm«nl* I 

2 How .lfliiy children in your apart im'ni ;utcnd Lincoln Unified School Ijisinci > 

l Identity tndividunl schools by name* 2. 

3. Htm niftny adulu currcnlJy mend GAJN program bngluh classes? 3 

4. When- do ihey uiwnd classes* (e.g.. Dclu College, adult ^hiK>l I 4. 

5- ll (here arc adults living in this upartitium mho are am attending GAIN Kiidn-h 

classes, would they attend GAIN biiglkh classes if they were held in your 

aponmcni complex .' 5. 

6. How muny men would go'' d 

7. How many wtimcn would go? 7. 

H. Would Ihcsc adults Jprec to attend cla>s Tour hours u duy. five days j week ? b 

9. Would ihcy need lo hu s e someone outside the family lake care ol their 

clntdrcn during class? *> 

IU. Haw many children need child care * Itf 

1 1 . Should clats he held in the morning or in the aitcmiKin 1 1 1 

12 lb il u good idea Tor us to iry to get otir classes appnued Tor GAIN, vir should 

wc huvc our own literacy program • 12. 

13. If wc cannot gel our English claues- »ppro\ed Tot GAIN, would you unend 

anyway? 1 3. 

14 Huw nun > hnur\ per da> should classes beheld 1 14. 

1 5. How many day* per week should classes be held'* 15. 

16. Whal would yuu like lo leani? In 

a. Kn&lish d. Huw Ihe schools operate 
h. How u> gel n job c How lo become a sili/cn 
c Huw to oe a govd puwi 

V- Whal rs your level nl hngkfih lrtnt»ua?c skills ' 

a Oral skills- I 1) Luw <2l Medium <3l High 17a 

b. Reading and wntmg shUV (1 1 Uwv I J) Medium iJiHigh 17b. 

1 8. How long have you been m ihe United Stales ■ 18 

I*}. How many children in your opaninuni arc pamcipaimg in BSEA Tide VII priigrami.' t9. 



Family English Literacy Project • Lincoln Un Mod School District • Glockton. California 



daia have been compiled, the lotal frequency count should equal ihe 
number of people who responded to the survey. 

Analyzing Survey Data 

The results of the needs assessment can be analyzed to answer 
questions related to planning or evaluating the project. The process of 
analysis consists of reviewing the compiled data and making judg- 
ments about the needs of the learners or about how the project can be 
improved. Staff may find it helpful to organize the data analysis by 
posing questions that focus on key aspects of the project. Figure 1 
contains sample questions thai staff members might use for organiz- 
ing, analyzing, and using data that are collected with a survey. The 
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corresponding questions from the needs assessment survey are 
identified in parentheses. 



Staff may find it helpful 
to organize the data analysis 
by posing questions that focus 
on key aspects of the project. 




Sample Questions for Data Analysis 



1 . How many adults living in this community (male and female) are 
eligible to participate In the project but are not enrolled currently 
in English-as-a-sGcond-language (ESL) classes? (questions 1, 
3. 6, and 7) 

2. How many adults living in this community are interested in 
attending ESL classes? (questions 6 and 7) 

3. What are current and potential participants' needs for child 
care? (questions 9 and 1 0) 

4. What are current and potential participants' preferences 
regarding the scheduling of classes? (questions 8,11,14, and 
15) 

5. What are current and potential participants 1 preferences 
regarding 1 the content of instruction? (question 16) 

6. What are current and potential participants 1 speaking, reading, 
and writing skills? (questions 17a and 17b) 



Given the high turnover 
of participants in many family 
literacy projects, a survey such 
as the needs assessment 
survey should be conducted on 
a regular basis to ensure that 
project services are based on 
current learners' needs. 



Staff Use of Survey Data 

Survey data may be used by staff to determine how well the 
project is meeting the learners' needs. Given the high turnover of 
participants in many family literacy projects, a survey such as the 
needs assessment survey should be conducted on a regular basis to 
ensure that project services are based on current learners' needs. For 
example, the findings from Questions 1-5 can be used to identify 
important background information about the participants. This 
information also can be used to help staff determine the effectiveness 
of their recruitment efforts in various neighborhoods. In addition, it 
can help staff evaluate the suitability of current class locations and 
identify more convenient sites such as community centers or apart- 
ment complexes. 

During regular project meetings, staff can consider the results of 
surveys to determine the extent to which they are addressing the 
learners' needs and to deal with other issues that might have been 
identified when the survey was conducted. Such discussions enable 
staff to improve current services and to plan future activities. Survey 
data, collected annually or more frequently, can be used to answer 
questions related to developing annual evaluation reports (for 
example, "What needs have been met?"; "What new needs have been 
identified?"; "What progress has been made in meeting project 
objectives?"). 
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During regular project 
meetings, staff can consider 
the results of surveys to 
determine the extent to which 
they are addressing the 
learners' needs and to deal 
with other issues that might 
have been identified when the 
survey was conducted. 



The reporting of survey 
results should focus on how 
staff can improve the project. 



Reporting Survey Results 

Methods of reporting the results of surveys depend on how the 
data will be used. If the results are to be used for project evaluation, 
they could be organized around the evaluation questions shown in 
Figure 1 in the preceding section. The data corresponding to each 
survey question may be ( 1 ) described in brief sentences, for example. 
"More than half (55%) of the respondents indicated that they need lo 
have someone outside the family take care of their children while they 
attend class" or "Approximately 30 children will need child care" or 
(2) illustrated in tables depicting the range of responses and number or 
percentage of responses. Figure 2 illustrates how data from Question L 
of the needs assessment survey may be presented in a table for a needs 
assessment or evaluation report. 

The reporting of survey results should focus on how staff can 
improve the project. The conclusions or recommendations should 
provide answers to questions that staff and learners have posed about 
the project. Figure 3 contains a table and narrative that summarize the 
responses to Question 3 in Figure 1 ("What are current and potential 
participants' requirements for child care?") and present findings and 
recommendations. 

Performance Samples 

Performance samples are illustrations of learners' work in 
selected tasks. As one kind of performance sample, a writing sample 
can provide valid, authentic representations of learners' progress in 
attaining project objectives. Because they can be linked directly to 
project activities, writing samples become natural extensions of the 
instructional process. Learners can produce writing samples in 
English, their native language, or both, depending on the purpose of 




"Number of Ac Jits Living in One Apartment 



Number of 
Adu'ts 


Number/Percentage 

Responses 
Total Responses=25 


3 


8 (32%) 


4 


1 1 (44%) 


5 


6 (24%) 



The siwey results indicate that more than half (68%) of the 
respmdenteJiye to ag^ments housing Jour DrJweLeidults. - 
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the project and their literacy ability. As learners develop examples of 
their written work throughout the year, they and the staff can identify 
writing samples that demonstrate strengths as well as areas that need 
improvement. 

This section presents strategics for collecting, analyzing, using, 
and reporting the results of alternative approaches to assessing writing 
samples. Strategies refer to the writing assessment instrument shown 
in Chapter IV, Example 10 (reproduced on the following page). 

The writing assessment instrument uses a rating scale to evalu- 
ate the learner's performance. A rating scale is a technique for assess- 
ing an area of interest (for example, reading or writing ability or self- 
confidence) on a scale of intervals usually designated by numbers or 
descriptors or both. Rating scales are frequently employed to differen- 
tiate performance levels. Ratings are based on the learner's perfor- 
mance relative to narrative explanations (descriptors) of each indi- 
vidual rating on the scale. Numerical values can be attached to the 
descriptors to facilitate data analysis. Sometimes ratings are combined 
with narrative comments and placed in a performance chart to provide 
for a more comprehensive analysis (see Example 9 in Chapter IV). 
The writing assessment instrument in Example 10 enables project 

















! 

! 



Summary of Respondents' Child Care Needs 



Item 

Number of adults 
who need child care 
to attend class 


Currently Serving 
25 


Need to be 
Served 

35 


Conclusions 

Ten additional adults ret,, re child 
care services. 


Numbei of children of 
adult participants 


30 


45 


Child care facilities are currently 
filled to capacity. Staff need to 
locate additional space for children 
of new learners. 



The original survey identified 25 adults who needed child care services in order to attend 
classes. After four weeks, 1 0 new learners requested child care services in order to attend class. 
Because the need for child care Is increasing, staff recommend that additional space be located for 
children of new learners. 
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Example 


Writing Assessment Instrument 


A. AulhenliciU /vuiCc/cngagemcnl of (he reader 4 = foprcsMun sinwip ly rcHceu Utar writer h cmu- 

Iwnal and Air imcHcffuaf invoKi-mCm in [Iil- lopic 
Sir imply ciigzga. Ihc alicntum uf ihc reader. 

Sew ._ _ 

1 ■ Tlic vsTilrr i<i encaged in ihc ionic and ciivaers ihc 
Cummenii ruder. 




2 k Uninteresting: nol engaging; perfunctory 




1 = \\ nunp wini lo be mechanical cuen-lic. 
Marked hy cliches. ha/> generalization*, mean- 
injtlcs enPrctoiunv 


B h*uv'nr>Mni/jimru'Jeirl<tprneni 


4 s hocuM-i mi nnr main 'ilea Has clear bceirmin«. 
middle . and end V\ i-ll urf.im fK d and "Acll 
devclupcd ihmugh example*. 


Scare 
Comments: 


3 - tixuud itnd organize d hul mjy lunc n (law in 
coherence or mcnmpleic closure. Incomplete 
dcvclnpmenl. Explanation i* urongly implicit 

2 ■ Lack uf clear locus, organi/alion. or Jc\i'kip 
mem Narrative, hm noc^plunjnon 

1 > Disorganized, nndcrdci eloped, unconnected 
genera ii/2i ion v 


f Sentence mcchanrcv/langusigi: 


4 ■ Pew mechanical, usage, or *emc nee error*. 
Language lived wuh fluency and \nncty 


Score 

CffmNcitf*' 


X - Sor »■ mi run mcehanical usage nr sentence errors 
I-anguagc uud competently (u express, idea* 

1 = Hnuugh u^apc errors io jucaci .iiicmton auay 
fmm ihc cimi';n1 Sentences understandable, hul 
unconvcm lOnnl. 

1 = 1 -anguagc and mechanical errors nnpnir liwanine 



Source AuujMed from Rcna Soifcr ct al ( I Will The t twiplvt* tfaon. m pimiu e favulbaik of tlituli hitrat » 
Curm ii/um r/i-ifgn ami trat tofiff uflpnWM+f « New York I cat hers College Press, pp. 1 715 1 7«J. 



staff to rate the learners 1 writing skills in each of three traits, or dimen- 
sions, of writing: authenticity, focus, and mechanics. Each of these 
traits is rated on a four-point scale, from poorer skills ( I ), to strong 
skills (4), according to descriptors provided for the ratings. 

Staff members could adapt the instrument by developing descrip- 
tors that would be appropriate for the learners and curriculum in their 
project. By developing project-based descriptors and associated rating 
scales, staff can improve the validity and reliability of the instrument 
and enhance the accuracy and consistency of assessment and data 
collection. The descriptors should be arranged in logical sequence with 
clear definitions for each rating. These are complicated tasks, and staff 
may wish to seek outside assistance. 

As a natural extension of a learning activity, a writing sample 
serves both instructional and assessment purposes, thereby eliminating 
the time and energy required for separate tests. By collecting and 
assessing writing samples on a regular basis and with agreed-on rating 
scales, staff members can make judgments about learners' growth and 
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By developing project- 
based descriptors and 
associated rating scales, staff 
can improve the validity and 
reliability of the instrument 
and enhance the accuracy and 
consistency of assessment and 
data collection. 



As a natural extension 
of a learning activity, a writing 
sample serves both 
instructional and assessment 
purposes, thereby eliminating 
the time and energy required 
for separate tests. 



about progress learners are making toward attaining the objectives of 
the project. To use rating scales effectively, staff members need to 
have a thorough understanding of the descriptors. Staff members can 
achieve this understanding by receiving training in the rating proce- 
dure and by collaborating to design the descriptors for the ratings. 

Collecting Data from Writing Samples 

Figure 4 shows how data from learners' writing samples can be 
collected systematically. Data obtained through this procedure can be 
used to monitor learners* progress in attaining the objectives of the 
project. 

The class list helps staff record assessment results of writing 
samples on a regular basis, such as weekly, monthly, or quarterly. 
Staff can use the rating scale to judge learners' writing or ask learners 
to judge their own work. To accommodate the learners' language 
proficiency levels, staff can simplify the descriptors so learners can 
read and use the rating scale for their self-assessments. In this way, 
assessment becomes an instructional tool that enables learners to 
recognize and judge effective writing. Portfolios can be used to keep a 
record of selected examples of learners' writing. Although learners 
may write on a daily basis, staff can have learners select their best 
work for inclusion in the portfolios on a specified schedule. 

Analyzing Data with Frequency Counts 

Simple frequency counts provide a useful way to analyze data 
collected with rating scales. For more sophisticated data analysis, staff 
should seek the assistance of a professional evaluator. The analysis of 
ratings of writing samples can be facilitated by using a class list such 
as the one shown in Figure 4. Used wilh the writing assessment 
instrument, the list helps staff document the ratings according to the 
four descriptors (1-4) in each of the three dimensions of writing: 
authenticity, focus, and mechanics. The list provides space for ratings 
taken at three different times — first testing, second testing, and third 
testing. This format enables staff to monitor the progress of each 
individual as well as the total class. 

The class list contains the ratings of a partial number ( 10) of 
participants in a family literacy class. The data represent the three 
assessments conducted in October, January, and April. The numbers 
under A, B. and C represent the assigned ratings according to the 
descriptors specified in the instrument. The list can be used internally 
by the staff or provided as a table in an annual evaluation report or 
final performance report. 

The data in the class list may be analyzed further by counting 
how many learners received each rating for each aspect at each data 
collection point. The tallies arc converted to percentages because the 
number of learners generally varies from one testing interval to the 
next. Staff can interpret the impact of the project by examining the 
degree of change in percentages across the three testing intervals. This 
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Portfolios can be used 
to keep a record of selected 
examples of learners' writing. 
Although learners may write 
on a daily basis, staff can have 
learners select their best work 
for inclusion in the portfolios 
on a specified schedule. 



change can be illustrated graphically and documented in a matrix such 
as the one shown in Figure 5 (see page 102). 

Figure 5 provides a summary of the results of writing assess- 
ments conducted at three testing intervals (October, January, and 
April). It displays ratings for three writing traits: authenticity, focus, 
and sentence mechanics. If this summary were included in an evalua- 
tion report, it would be accompanied by a narrative description of the 
progress and trends toward accomplishing project goals; 

The table reflects an increase in the percentage of learners who score at 
higher levels with each assessment, suggesting progress in attaining 
project objectives. Tiie greatest growth took place in authenticity. 

Using the Results of Assessments of Writing Samples 

Staff can use the results of writing assessments for planning 
instruction, monitoring progress, determining the degree to which 
objectives have been achieved, and refining the curriculum and 
instruction. The results can help staff members form groups of learn- 
ers with similar needs and design appropriate instructional activities. 
Data can be analyzed to indicate the progress of individual learners as 



figure 4 



! Class List: Results of Assessments of Writing Samples : 

Instructions: Record date of the assessment and enter the rating (1-4) under each letter ! 

A = Authenticity/Voice /Engagement of the Reader, B = Focus/Organization/Development; j 

i C = Sentence Mechanics/Language j 

i ; 



Name orl.D. No. 


Date 


First Testing 


Date 


Second Testing 


Date 


Third Testing 


1 — ■ j ■ 

Affile 


A 


B 


C 


A 


B 


C 


B. Lopez 


10/25 


r ' 

1 i 1 ■ 2 

i 1_ , .-4~ 


1/15 


1 


2 


2 


4/2 


1 


2 


3 


J. Garcia 


10/15 


1 1 : 1 


1/15 


1 


1 


1 


4/2 


1 


1 


1 


M. Matello 


10/15 


r « — * 

1 j 1 j 1 


1/rs 


1 


2 


2 


4/2 


1 


2 


3 


K. Turner 


10/15 


1 


2 j 3 


1/15 


2 


3 


4 


4/2 


2 


1 


2 


R. Diaz 


10/15 


2 


1 


2 


1/15 


2 


1 


2 


4/2 


2 


1 


2 


B. Ruiz 


10/15 


1 


2 


1 


1/15 


2 


2 


2 


4/2 


2 


2 


3 


A. Rodriguez 


10/15 


1 


1 


1 


1/15 


2 


1 


2 


4/2 


1 


3 


3 


J. Fernandez 


10/15 


2 


1 


1 


1/15 


2 


2 


2 


4/2 


3 


4 


3 


C. Lopez 


10/15 


1 


2 


1 


1/15 


2 


4 


2 


4/2 


2 


4 


3 


B. Nunez 


10/15 


3 


4 


3 


1/15 


3 


4 


4 


4/2 


4 


4 


4 
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Staff may choose to 
involve the learner In 
developing descriptors, 
selecting writings to be 
assessed, and ultimately 
evaluating their own writing. 



well as trends of the whole class. Assessment results also can show 
the relative effectiveness of various components of the project. 

Staff may review several writing samples by one learner to 
identify changes and trends in the learner's writing strengths. Such 
analysis can be accomplished using formal rating scales of writing 
traits, holistic scores, or without any type of actual rating. Learners' 
writing samples may also be shared among staff to gain insights into 
the overall effectiveness ot instructional activities. When learners are 
promoted to higher levels, their writing samples can be passed on to 
their instructors to ensure that instruction is appropriate and 
uninterrupted. 

Writing assessments are instructional tools as well as vehicles 
for appraising learners' progress. When shared with the learner, the 
results provide immediate feedback that can lead to improved writing 
performance. Staff may choose to involve the learner in developing 
descriptors, selecting writings to be assessed, and ultimately evaluat- 
ing their own writing. Such self-assessments can help learners acquire 
an awareness of the characteristics of effective writing. 

When parents appreciate the importance of writing and other 
literacy behaviors, they can become more confident in supporting 
their children's language development. When parents experience 
success at writing creatively about meaningful topics, they can 
transfer their joy of writing to their children. They may share their 
writing with their children, encourage their children to write about 
their own experiences, and even write stories together. 



Summary of Three Writing Assessments 



Testing 


Authenticity 


Focus 


Sentence Mechanics 


1 


2 


3 


4 


1 


2 


3 


4 


1 


r\ 


3 


4 


First=Oct. (n=125) 


66% 


22% 


12% 


0 


68% 


20% 


12% 


0 


80% 


18% 


2% 


0 


Second=Jan. (n=H8) 


2% 


38% 


34% 


26% 


14% 


44% 


25% 


17% 


43% 


24% 


22% 


11% 


Third=April (n=105) 


0 


17% 


29% 


54% 


0 


17% 


49% 


34% 


20% 


53% 


23% 


4% 
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When parents 
experience success at writing 
creatively about meaningful 
topics, they can transfer their 
joy of writing to their children. 



Summarizing Writing Sample Assessments 
for Evaluation Reports 

Methods of reporting the results of writing assessments vary 
according to the purpose of the reports. One approach may be to 
present the class list (Figure 4) with an accompanying narrative in an 
evaluation report. The report could be part of an application for a 
continuation grant or used by project staff to plan additional classes 
and services. The data summarized in Figure 5 could be included in a 
final performance report. To provide an interpretation of the data, staff 
can develop a narrative that summarizes changes that resulted from 
the project and progress made by the learners in achieving the objec- 
tives of the project. 

Figure 6 illustrates how staff might report the results of three 
assessments of \ Jng samples. The example, containing a table and 
narrative explanation, is based on the writing assessment instrument 
(Chapter IV. Example 10). 




Learners' Growth in Writing (Authenticity) 





Authenticity 


Testing 


1 


2 


3 


4 


First Testing (Oct.) 


66% 


22% 


12% 


0 


Second Testing (Jan.) 


2% 


38% 


34% 


26% 


Third Testing (April) 


0 


17% 


29% 


54% 



Sample Narrative 

The table identifies the percentage of project participants 
scoring at each level (1 , 2 t 3, 4) for each of the three assessment 
intervals (October, January, April). The percentage of learners 
who made progress from low levels (1 and 2) to high levels (3 
and 4) Increased substantially over the three testing intervals. 
The initial assessment in October identified 66% of learners at 
Level 1 and no learners at Level 4. By the second testing In 
January, however, only 2% of the learners scored at Level 1 and 
26% scored at Level 4. By the end of classes in April, the third 
interval testing indicated that no learners scored at Level 1,17% 
scored at Level 2, 29% at Level 3 t and 54% at Level 4. Learners 
participated in an average of three hours per week of writing 
Instruction and associated activities for a total of 72 hours during 
the 24 weeks of classes. The assessment results demonstrate 
that learners have made progress in writing content that is 
authentic and engaging. 
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Interviews 



Because interviews can 
be conducted in the learners' 
native language, participants 
may be more willing and able 
to provide information to staff. 



Interviews can be used to collect detailed information for 
assessing and evaluating the project. An interview procedure may be 
used with individuals or small groups of people such as focus groups. 
This section will deal with the collection, analysis, staff use, and 
reporting of data obtained from focus groups. 

Interviews are particularly appropriate for family literacy 
projects because participants may not have the literacy skills neces- 
sary for paper-and-pencil assessments. Furthermore, because inter- 
views can be conducted in the learners' native language, participants 
may be more willing and able to provide information to staff. 

Interviews may be used independently or together with other 
measures to generate data to evaluate the effectiveness of instructional 
programs and to identify successful components of the project. They 
can be conducted by an instructor, project coordinator, or professional 
evaluator. This section of the chapter is based on interviews con- 
ducted with small groups — focus groups — according to a protocol 
that focuses on specific topics. Protocol is a specified set of proce- 
dures and questions designed to achieve the purposes of the assess- 
ment activity. 

Collecting Interview Data Through Focus Groups 

Focus group data can be collected with a qucstion-and-answer 
procedure in which a facilitator addresses questions to group mem- 
bers. The responses are documented in writing or recorded on audio- 
or videotape. Although the proceedings may be recorded by the 
facilitator, staff members may want to assign this responsibility to a 
separate individual to avoid burdening the facilitator. An important 
element of the focus group is the nature of the communication (verbal 
and nonverbal) between the facilitator and the respondents as ques- 
tions* answers, and comments are stated and clarified. Therefore, the 
facilitator should be sensitive to the cross-cultural and other interper- 
sonal dimensions that characterize group activities. 

The protocol for a focus group should be established in advance 
and linked carefully to the goals and objectives of the activity. (See 
Figure 7, p. 109, for an example of a protocol.) However, the facilita- 
tor should not discourage responses that may deviate from the main 
purpose of the session. Such flexibility creates opportunities for the 
facilitator to assess a variety of related learner needs and outcomes 
such as language, social interaction, confidence, and critical thinking, 
that are very difficult to assess through paper-and-pencil measures. 
These issues may be recorded in anecdotal form and analyzed later 
when the tapes are reviewed. 

Interview data may be collected at intervals throughout the year, 
on a pre/post basis or at the end of instructional units. The data can be 
analyzed and interpreted in terms of longitudinal progress, pre/post 
changes, and project implementation and accomplishments. Gener- 
ally, focus group data are collected and reported on the entire group, 
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although data gathered on individuals during the session also may be 
useful. 



Interviews are 
particularly appropriate for 
family literacy projects 
because participants may not 
have the literacy skills 
necessary for paper-and-pencil 
assessments. 



An important element 
of the focus group is the 
nature of the communication 
(verbal and nonverbal) 
between the facilitator and the 
respondents as questions, 
answers, and comments are 
stated and clarified. 



Analyzing Focus Group Data 

Analyzing focus group data can be a challenging task. As ihey 
review large amounts of data, staff need to deeide which findings to 
analyze in detail, which ones to mention only briefly, and which ones 
to ignore altogether. The importance attached to specific findings is 
determined by the evaluation questions, project objectives, and the 
intended audiences (for example, school board, project staff, and 
funding source). Ultimately, decisions regarding the complexity of the 
analyses should be guided by how the results are going to be used. 
Thus, the analysis could focus primarily on addressing the informa- 
tional needs of program decision makers, yet be sufficiently flexible to 
identify issues that were not anticipated. 

The goal of analyzing focus group data is to examine systemati- 
cally the participants' comments in order to identify trends or patterns 
that recur among different individuals or groups (Krueger, 1988). The 
outcome of the analysis may be a report that summarizes relevant 
trends, presents conclusions about the project's effectiveness, and 
offers suggestions for improvement. To accomplish these tasks, the 
analyst must sort through an array of statements and identify opinions, 
feelings, or perceptions that are common to several participants, even 
though they may be expressed in different words or styles. Some 
attention needs to be paid to determining the range and diversity of 
participants' perceptions. Less emphasis is given to singular opinions 
not supported by others in the group. The use of numbers or percent- 
ages may help illustrate participants* responses, although they are not 
neeessary for describing trends or supporting conclusions drawn from 
the analysis. 

A common misconception is that the analysis of focus group data 
is a simple process of finding a few good quotes thai support the 
analyst's point of view. To produce valid conclusions, however, the 
analysis needs to be conducted systematically according to a pre- 
scribed sequence of procedures (see the three- stage process below). 
The process should allow staff to analyze a large amount of qualitative 
data and draw practical conclusions about the effectiveness of the 
project. 

Ideally, the person who facilitates the focus group should also 
analyze the results. The results ean be interpreted in part based on the 
firsthand experiences of the facilitator who actually witnessed the 
interpersonal dynamics, body language, and affective factors that 
influenced the activity. Sometimes, however, it may not be possible 
for individuals who collected the data to be involved in the analysis. 
For example, an instructor may conduct an interview with a small 
group of learners with the assistance of a bilingual staff member. 
Audiotapes of the interview, translated into English if necessary, 
might be reviewed and summarized by a professional e valuator for an 
evaluation report. Comments by those who conducted the focus group 



115 



Chapter V fti 



BEST COPY AVAILABLE 



The goal o* analyzing 
focus group data is to examine 
systematically the participants 1 
comments in order to identify 
trends or patterns that recur 
among different individuals or 
groups. 



could be added to the transcript, providing the evaluator with valuable 
additional information. 

According to Krueger (1988), the process of analyzing focus 
group data can be divided into three stages: ( 1 ) reviewing the raw 
data; (2) creating descriptive statements; and (3) interpreting the 
results. 

Reviewing the raw data. At this stage, staff members analyze the 
tapes or notes taken during the focus group discussion. The raw data 
consist of the participants' actual statements recorded in writing or on 
audio- or videotape. Also included in the raw data are the facilitator's 
descriptions of the participants, such as their attitudes and body 
language, and the general ambience of the session. 

During this stage, the analyst should focus on becoming as 
familiar as possible with the range of data collected and then begin 
organizing participants* responses by categories or topics. Data may 
be organized by ( 1 ) marking all related topics in the transcript with the 
same colored marking pen; (2) extracting comments that deal with the 
same issue and writing them on a single page; or (3) rearranping the 
text according to specific categories. 

Creating descriptive statements. The next stage in the analysis 
involves reducing and organizing the raw data by creating descriptive 
statements that summarize the participants' responses according to 
topics. The summary also can include illustrative quotes that are 
especially insightful or indicative of the participants* comments. 
Comments need not be U/anscribed verbatim, rather, staff may prefer 
to use alternative statements that capture the speakers* intended 
meaning. The analyst should begin reducing the data and preparing 
the descriptive summary as soon as po'-siblc after each focus group 
session. 

Interpreting the results. At the final stage, the analyst needs to 
enhance the descriptive statements by drawing conclusions and 
interpretations from the data. The analyst should form conclusions 
about trends or patterns based on the purpose of the focus group 
session (for example, answers to staff's evaluation questions or 
evidence of progress toward meeting project objectives). Unexpected 
outcomes of the session also should be noted. To draw valid infer- 
ences from the data, the analyst should consider such factors as 
commonly used words or phrases, the frequency of certain responses, 
and the tone and intensity of the speaker's voice. It is also important to 
examine the context of part icipants* responses — that is, whether the 
comments occurred in response to a specific question or were sponta- 
neously triggered by an open-ended question. 

The analyst may give more weight to comments based on some 
criteria; for example, statements based on personal experience may be 
accorded more importance than something the speaker heard someone 
else say. Other factors to consider are the consistency of particular 
statements and whether individuals change their minds based on 
others' comments. Analyzing such faclors will enhance the validity 
and reliability of the findings. 
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Using Focus Group Data 

Staff can use focus groups to gather information that is not easily 
obtained through written assessments. The interactive nature of the 
focus group process allows staff to shape the direction of the session 
and to delve into selected responses. By probing specific topics, the 
facilitator can uncover causes of a problem and identify potential 
strategies for refining and improving the project. Focus groups enable 
staff to explore aspects of the project from the participants' perspec- 
tives, thereby enhancing the linkage between the learners' needs and 
the project's objectives. 

Focus group data can be used to identify new needs of partici- 
pants as well as clarify needs that were identified earlier through sur- 
veys or individual interviews. By learning more about participants — 
their attitudes, the cultural barriers they face, their patterns of social 
interaction, and their levels of confidence — staff can make iuore in- 
formed choices about curriculum content, instructional approaches, 
and the overall design of the project. 

To use focus group data effectively, staff need to identify trends 
and patterns among the group's responses. In general, the significance 
of a given issue is determined by the frequency of comments about it. 
For example, a single comment such as "The book is too hard" would 
not carry as much weight in the data analysis as five or ten statements 
about the difficulty of the instructional materials. If several comments 
were made, the staff might want to examine how appropriate the 
materials are for the learners' literacy levels. 

Staff also can use focus groups to obtain information about 
literacy -related events in learners' families. For example, a prompt 
such as "Tell me how you are using the home learning kits with your 
children" may help staff determine the effects of project activities on 
the learners' families. Comments may range from "Using the materi- 
als weekly and enjoy them very much" (ten respondents) to "Not 
using the booklets because we don't have any crayons" vtwo respon- 
dents). The first response helps confirm the success of the project; the 
second suggests that the staff need to help some parents obtain what is 
necessary to use the home learning kits. 

Using focus groups enables staff to generate more detailed 
information than is possible through some other assessment proce- 
dures. For example, parenting skills, attitudes, confidence, and other 
hard-to-assess domains can be explored through focus groups. Be- 
cause feedback is gathered directly and spontaneously from the 
participants, results are often highly indicative of the participants' 
perspectives. 

Summarizing Focus Group Data in a Narrative Report 

Reports of focus group interviews shoula be designed according 
to the needs of the audience, the purpose of the report, and the goals 
and objectives of the project. The results of focus groups can be 
communicated oraily or in writing. According to Krueger ( 1 988), 
focus group reports can be written following three different models: 
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1 . Raw data model thai presents the question or topic that the focus group 
addressed and a list of the participants 1 responses. 

2. Summary description model that summarizes the results of the focus group 
and includes selected quotations from participants. 

3. Interpretive model that provides a descriptive summary, illustrative 
quotes, and a narrative interpretation of the results. 

The raw data model is the fastest and easiest to develop and is 
most appropriate for an audience that is interested in reviewing all the- 
participants' responses, This model, however, requires that readers do 
their own analysis of the data. The descriptive and interpretive models 
are usually more effective because they provide examples of the data 
as well as detailed analyses of the findings. 

A focus group report should include a statement describing the 
purpose of the session, an outline of the key questions addressed by the 
group, and a summary of the way the session was set up (for example, 
number of people interviewed and method of selecting participants). 
The results of the report should be organized and presented according 
to the key questions that the focus group addressed. It may be helpful 
to describe aspects of the session that may limit the usefulness of the 
data or prevent conclusive statements about the project. A section for 
conclusions and recommendations is useful for integrating the key 
findings into a concise summary and providing suggestions for modi- 
fying the project based on the results. An appendix to the report may 
include the focus group protocol, quotes from the participants, or the 
entire transcript. 

Figure 7 shows how to document the purpose, protocol, and re- 
sults of a focus group session that addressed parenting workshops held 
during a family literacy project. The sample narrative might appear in 
an annual evaluation report or final performance report. 

Observation Measures 

Observation measures can be used to collect and record various 
kinds of information. They may focus on such features as learners 1 
characteristics, group interactions, or language and literacy perfor- 
mance. Observations may be used as closed-ended assessments, such 
as indicating the learner's level of listening comprehension, or they 
may be more open-ended, as when staff members judge how confident 
a learner appears when participating in a cooperative learning group. 

This section deals with the collection, analysis, staff use, and 
reporting of data obtained with observation measures. References are 
made to the measures that appeal" in Chapter IV to assess listening, 
reading, speaking, and writing, A modified cohort design is presented 
to illustrate how projects can analyze assessment data in a manner 
consistent with evaluation requirements for comparison group data. 
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Figure 7 



Sample Stjmroary of Focus Group Results 

The purpose of the focus group was to determine the 
effectiveness of the five parenting workshops held on 
Saturdays and the degree to which the workshops had met the 
needs of the adult learners and the objectives of the project. 
Thirty-five learners participated in the focus group. 

Protocol for the Focus Group 

Staff asked the parents to describe the parenting workshops 
they attended. Questions included: Describe how you were 
involved in selecting the topics. Were you unable to attend any 
of the workshops? Why? Do you feel you are more confident as 
a parent because of the workshops? Why? Do you help your 
children with their homework? What did you learn that you did 
not know before? What are you doing differently as a result of 
the workshops? 

At least half of the parents attended three or more of the 
five parenting workshops. They reported that they had an 
opportunity to work with staff to plan the workshops but were 
unable to recall any specific topics they had requested. All 
agreed that the workshops helped them as parents and 
enabled them to work more confidently wftn their children at . 
home. A majority of parents indicated that they reviewed 
homework assignments, although only a small number actually 
helped their children with reading and writing assignments. 

The parents expressed general agreement about the III 
effects of television viewing and indicated that they had been 
successful in reducing the amount of television that their 
children watched at home. Most stated that they 
communicated with their children in their native language, 
whereas only two parents indicated that they interacted with 
their children In English on a regular basis. The staff noted that, 
compared with the focus group held earlier in the year, the 
parents in this session seemed more communicative and 
confident when discussing parenting issues. 

In summary, the parents enjoyed the workshops and 
enthusiastically volunteered information during the sessions. 
Staff noted that parents who participated in both focus group 
sessions appeared more competent and confident when 
expressing their ideas in English than those who attended one 
session. Approximately half the participants could respond in 
English, and the other half required the assistance of an 
interpreter. In general, the workshops were well attended and 
seemed to provide parents with information that made them 
feel more confident about helping their children at home. Staff 
agreed thai future workshops could be more effective by 
dealing with topics that are (1 ) related to parent-child learning 
activities at home and (2) linked to the curriculum used in 
weekly family literacy classes. 



BEST COPY AVAILABLE 



119 



Chapter V fE 



BEST COPY AVAILABLE 



Collecting Observation Data with Rating Scales 

The observation measures shown in Chapter IV, Examples 7 and 
8 ( reproduced below) can be used to collect data about the learners' 
progress in acquiring receptive (listening and reading) and productive 
(speaking and writing) language skills. The instruments use rating 
scales to indicate the learner's degree of proficiency. 



' Observation measures 
can be used to collect and 
record various kinds of 
information. They may focus 
on such features as learners' 
characteristics, group 
interactions, or language and 
literacy performance. 



Example© 



Observation Measure for 
Listening and Reading 



Tjpcof iwanflRi Observation measure. 
Purpose: To assess liMCium: and reading skills. 



Pecoptrve skills 
U s t e n i n g Follows f ami 1 1 ar i nslruclion s 

Follows new instructions 



support 



VWft great Not at 
difficulty ad 



Peading Roads I a miliar instruction* 
Reads new mstructions 



Example© 



Observation Measure for 
Speaking and Writing 



Tjrpcor augment: Ub^naiiori measure 
Purpose; 1o dMCA spooking und wming skilk. 



Winsome 
support 



WHhgmat Not at 
difficulty $ff 



Speaking Produces lamiliar spoken 
instructions lhal are 
clear and comprehensible 



Produces new spoken 
instructions lhal are clear 
and comprehensible 



Produces lamiliar wrtren 
Instructions that ere ctser 
and comprehensible 



Produces new wnlten 
msif ucifoos that ere dear 
and comprehensible 
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The data collected with the observation measures can be used to 
make statements about the performance of an individual or group 
according to descriptors that are associated with each of the ratings: 
With ease, With some support, With great difficulty, and Not at 
all. The instruments are depicted as examples that staff can adapt to 
reflect Ihe objectives of their project. Staff can modify the instruments 
by designing their own descriptors and establishing their own protocol 
for rating their learners in a consistent, reliable manner. 

The descriptors reflect staffs expectations about how learners 
will perform at various stages of development. The descriptors should 
provide clear distinctions among developmental levels on the rating 
scale and represent appropriate growth according to the project's 
literacy framework (see Chapters H and III for a discussion of the 
literacy framework). The lowest rating and its descriptor should 
reflect what the staff has determined to be the lowest level of language 
proficiency exhibited by new enrollees. The highest rating and its 
descriptor should indicate mastery or the highest level of proficiency 
the staff consider attainable for the learners in the project. Each item 
on the scale should be mutually exclusive and part of an even nrmber 
of selections (for example, four or six), thereby eliminating middle- 
ground options and requiring observers to make clear choices. 
Creating well developed descriptors and training staff to use the 
observation measures will enhance the reliability and validity of 
the instruments. 

Assessment through observation means watching learners 
perform various tasks related lo the project. Such assessment can be 
done without having to set aside additional time in the project for 
testing. The observations should be based on the project's objectives 
and conducted systematically and consistently at predetermined 
time intervals. 

The following is a summary of strategics that family literacy 
staff can use for obtaining accurate and meaningful data with observa- 
tion measures. 

1 . Establish descriptors that define different levels of performance that can 
be agreed on by similarly qualified raters who are familiar with the 
instructional design of the project. The descriptors should be (a) aligned 
with the curriculum and objectives of the project and (b) reflective of the 
lowest and highest performance levels of the learners. 

2. Create protocols and record observations at regular intervals (for ex- 
ample, monthly, quarterly, or annually). The frequency is determined by 
how the data will be used and the length of instructional units (lesson, 
course, or class session). 

3. Develop an instrument for collecting and managing the data such as the 
two observation measures shown in Examples 7 and 8. 

Methods for collecting observational data should ensure that the 
form for collecting the data is completed for each learner according to 
established intervals and that each learner's level of performance is 
recorded for each skill. The time intervals for assessment are 
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dcicrmincd by the techniques that will be used to analyze the results. 
For example, a pre/posl analysis requires that data be collected at the 
beginning and end of an instructional unit or specified period of lime. 
To do a longitudinal analysis, staff need to collect data at more fre- 
quent intervals and over longer periods of time such as a year or more. 
Collecting data at frequent intervals for longitudinal analysis helps 
staff idenlify patterns and trends in the learners' performance. Such 
data are useful for comparing the performance of the project partici- 
pants wuh individuals who have similar characteristics or who have 
not received project services. 

Analyzing Observation Data with a Modified Cohort Design 

Staff can analyze data obtained with the observation measures by 
reviewing the results and determining the extent to which learners are 
making progress in attaining project objectives. Staff may want to ana- 
lyze data separately for each language group or each English profi- 
ciency level served by the project. Based on their analysis, staff can 
modify instructional activities to enhance learner outcomes in the fu- 
ture. Additional analysis and interpretation can be conducted by as- 
signing numerical values to the descriptors on the rating scale. Staff 
can report how many learners received what rating at each prescribed 
interval. If the number of learners differs for each observation, the data 
can be presented as percentages of the total number of learners ob- 
served. Comparisons should be made through percentages, because the 
number of learners usually varies for each year of the project and for 
each collection interval. 

Data collected through observation measures, writing samples, 
and other types of alternative assessments may be analyzed according 
to a modified cohort design; that is, growth of project participants can 
be measured against a comparison group (cohort), thereby satisfying 
one of the requirements of many funding sources. This design is 
adapted from research by McConnell (1982). (See Chapter I for a dis- 
cussion of the comparison group issue.) 

The modified cohort design provides an alternative to comparing 
the project participants* growth with adult learners who are not part of 
the project. If the cohort design is implemented at the beginning of the 
first year of a literacy project, staff would establish comparison group 
data by collecting data on new enrol lees before instruction begins. As 
new learners enroll during the project, staff would add information 
about them to the comparison group data. At a predetermined interval, 
perhaps after completing an instructional unit, a semester, or (he first 
year, staff would collect observational data and measure it against the 
comparison group data to assess the learners' progress toward meeting 
project Sjectives. For an extended longitudinal analysis, assessment 
daia could be collected during the second or third year and measured 
against the comparison group data that had been compiled to that point. 

Some funding sources may require that part of the project evalua- 
tion be based on data collected at predetermined intervals, usually ev- 
ery twelve months. Staff can meet this requirement by assessing 
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project participants* performance after twelve months of instruction 
and comparing the results with the comparison group. Data on new 
enrollees who are added to the comparison group should be collected 
wiihin two weeks of enrollment, after the staff has become somewhat 
familiar with their background. The data become comparison group 
data, collected whenever new learners enroll during the year, thereby 
accommodating open enrollment policies adopted by many family 
literacy projects. 

The modified cohort design is based on the assumption that the 
comparison group data represent levels of performance that would 
have been demonstrated by individuals who had not henefited from 
services of the project — a population frequently referred to as a 
"nonproject comparison group."* 

After the comparison group data have been established, staff can 
compare differences between the comparison group and the project 
participants in terms of the number and percentage of individuals who 
received each rating on the observation measures. Differences between 
the two groups can be illustrated through bar or line graphs that iden- 
tify each group according to time in the project (see Figure 8, page 
117). Illustrations should be accompanied by a narrative that describes 
the relationship between the learners 1 performance and the services 
provided by the project. 

Examples 7 and 8 of observation measures, found in Chapter IV, 
contain two performance descriptors for each of the skills of listening, 
reading, speaking, and writing. By counting and reporting the number 
(or percentage) of learners observed performing at each level during 
each observation, staff can make global statements about the group's 
abilities in each skill. The observations may show that the learners 
were uniformly low in every skill at the beginning of instruction or that 
they exhibited great diversity in their abilities. Longitudinal or posttest 
data may show that most learners were performing at more proficient 
levels or that there was considerable variation in ability. 

The modified cohort design may show that learners who have 
participated in the project are performing at higher levels than mem- 
bers of the comparison group who have not yet benefited from program 
services, thereby validating the positive effects of the project. Longitu- 
dinal data collected over several years may reveal relationships be- 
tween the learners' growth in performance and length of time they 
were enrolled in the project (see graphic illustration of such relation- 
ships in Figure 8). 

Using the Results of Observation Measures 

To provide useful information, observation measures should be 
( I ) lied directly to the objectives and instructional activities of the 
project and (2) conducted on a regular basis. By linking the descriptors 
and progression of raiings to instructional priorities, staff can obtain 
valuable data for assessing learners" ongoing progress and for improv- 
ing the instructional program. The observation measures also can be 
used as inilial assessment tools for determining learners' needs. For 
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example, learners who are initially observed performing tasks with 
ease probably would need instruction that is different from ihose who 
are rated as performing them not at all. 

Staff can use data obtained with observation measures to modify 
curriculum content and instructional methodology. For instance, if the 
data indicate that many learners are performing listening tasks with 
ease but writing assignments with great difficulty, staff may want to 
devote more time and effort to writing instruction. Similarly, if 
several learners are observed performing a certain task with great 
difficulty even after it has been presented and practiced many times, 
staff may want to consider how instructional activities could be 
modified to help learners gain mastery of the task. 

To make the best use of data collected with observation mea- 
sures, staff members need to examine individual learners' perfor- 
mance and create aggregate profiles of the whole class. These findings 
can be discussed regularly at staff meetings to determine how the 
project can be more responsive to learners' needs. For example, 
learners whose individual data contain several ratings of not at all or 
with great difficulty may have special needs that warrant more 
personal attention. If the aggregate data show ratings that are spread 
across many performance levels, staff members may want to discuss 
how they can address multilevel classes more effectively. 

Individual learners' performance records may be used to deter- 
mine whether current instructional groupings are appropriate and how 
new groups might be formed. Significant variation in ratings may be 
due to discrepancies in the staff members 1 knowledge of how lo use 
the observation measures. Staff may need to clarify the protocols for 
using the measures and the definitions of the performance descriptors. 
Observation measures should be used systematically according to 
predetermined criteria that are related to the project's objectives. 
During the observations, however, staff may identify other relevant 
information that should be documented. For example, staff may note 
that some learners are frustrated, socially isolated, or have auditory, 
speech, or visual impairments. Staff also may document special 
talents and interests shown by learners. Such findings will help staff 
create activities that are more closely linked to learners" needs and 
make other improvements to the project. 

As with other alternative assessments, the purpose of observa- 
tion measures should be clearly established before an instrument is 
developed or used. Observation measures can be used to plan instruc- 
tion and to place learners in appropriate groups or activities. As 
discussed in Chapter IV, observational data can provide staff with 
information about learners' ongoing progress in attaining project 
objectives. Observation measures also can yield data for annual 
evaluation reports or the final performance report. 

Reporting the Results of Observation Measures 

Reporting observational data involves summarizing the results 
or developing interpretations of the observations. Data collected from 
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observational measures can be used to make descriptive statements 
about what individual learners or groups of learners can do relative to 
project goals, objectives, and instructional services. 

The observation measures found in Chapter IV are used to rale 
learners according to their receptive (listening and reading) and pro- 
ductive (speaking and writing) skills in handling new or familiar 
instructions. What is new or familiar depends on the learners' prior 
experiences and the topics addressed in the project. Thus, the reporting 
of observation data should include descriptions or examples of ihe 
content or activities lo which learners were responding when the 
observation ratings were made. 

Bar graphs are especially useful for presenting summaries of 
observational data. For example, each bar could represent the number 
of learners who performed a specific skill at a given level (that is, the 
number of learners who read new instructions with great difficulty ). 
The construction of the bar graph depends on what specific information 
the staff want to emphasize. Figure 8 illustrates growth in learners' 
ability over two years' time to produce written instructions that are 
clear and comprehensible. Figure 8 also illustrates the findings of a 
modified cohort design in which progress of learners who have partici- 
pated in the project for one and two years is measured against the 
performance of a companion group of learners who had not received 
project services. 

Figure 9 summarizes hypothetical data for a class of 25 partici- 
pants who were rated at quarterly intervals. The table is followed by a 
narrative explanation. The numbers and narrative address only one of 
the eight areas assessed by the two observation measures. Staff could 
develop similar illustrations for data obtained foi the other seven skill 
areas. When the numbers (w) vary greatly between groups of learners or 
years of implementation, percentages can be used instead of numbers. 

As was pointed out in Chapter I, open enrollment and uneven 
attendance in many family literacy projects often prevent staff from 
assessing the same learners every time an observation is conducted. 
Therefore, staff should explain this phenomenon in the evaluation 
report and present assessment results only for the learners who at- 
tended regularly. Staff also may want to include in the report some of 
the causes of attrition and the strategies that are being used to improve 
retention. 

If a project has open enrollment, a second quarterly observation 
conducted in one class may be the first observation for new enrollees 
and the second observation for experienced participants. In this case, 
data gathered on several learners should be aggregated only if each set 
of data represents approximately equal amounts of instructional l ime. 
Data oji learners who started at different times may need to be reported 
separately, or if all learners' data are combined, the first observation 
should in fact be the first for each learner. 

Each of the eight receptive and productive skills identified on the 
two observation measures should be illustrated and reported separately. 
A table that illustrated all eight skills might confuse the reader and fail 



125 



Chapter V 



BEST COPY AVAILABLE 



f igure 8 



Summary of Data Collected with Observation Measure (Modified Cohort Design) 

"Produces new written instructions that are clear and comprehensible" 



c 

A3 

£ 




WRh Ease 



With Some 
Support 



WBhGreal 
DHIkmlty 



Y/^ New enroltees (fte16) 

["""] Partkfcants lor one year (/>=30) 

S3S| Participants lor iwo years (n=20) 



Not At All 



Sample Narrative: Evaluation results based on a modified cohort design indicate that the project had a 
positive impact on the development of literacy skills among adults. Furthermore, the data suggest that 
learners who have participated for two years show substantially greater gains than those who have been 
in the project for only one year. Staff suggested that differences in performance between the two groups 
. were attributed in part to the learners 1 need for a full year of participation in order to make rapid progress 
in subsequent years. 




Summary of Ratings of Four Observations (n=ZB) 
Quarterly flaflfigs on "Prodiices new written instructions that are < 



Rating 


September 

First 
Observation 


December 

Second 
Observation 


March 

Third 
Observation 


June 

Fourth 
Observation 


With Ease 


0 


0 


3 


10 


With Some Support 


0 


5 


12 


8 


With Great Difficulty 


5 


5 


10 


7 


Not at Ail 


20 


15 


0 


0 



, JBampla Narrative: Figure 9 shows an Increass In the percentage of high ratings from the first to the fourth 
.observation and a decrease In the leas proficient ratinge. For the first observation In September, none of 
"the leamera was able to write new instructions that wp re clear and comprehensible, However, by the? fourth 
^observation En June, after nine months^ lflggj|g^ 
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to delineate important details. Staff should include a narrative inter- 
pretation of the results with each table as shown in the examples in 
this section, or they could describe the results in narrative form and 
place it with the related goal or objective as shown in Figure 10. 



Report of Learners' Progress in Attaining Objective 
- tor-Receptive and Productive Language Skills 

Objective -Literacy: By the end of each year of the project, 
participants will improve their receptive (listening and reading) 
and productive (speaking and writing) skills at least one level 
using observation measures developed by the project. 

Sample Narrative 

The objective was achieved during the second year of 
the project. Based on the observation measures, learners 
experienced growth in all eight literacy skill areas. The greatest 
growth occurred in listening skills with 72% of the learners 
rated with ease for that category. Fifty-seven percent received 
the rating with ease for speaking, 29% for reading, and 16% 
for writing. The results of the observation measures are 
consistent with the comments of staff who indicated that 
listening and speaking skills were emphasized during the first 
two years of the project in response to the learners' identified 
needs and requests. 

Based on the results of data analysis according to a 
modified cohort design, the project has had positive effects on 
the learners' literacy development. The language classes, 
which integrate literacy instruction with basic life skills, 
represent a potentially successful component of the project. 



Summary 

Alternative approaches to assessment and evaluation are espe- 
cially useful for meeting the needs of participants in a family literacy 
project. The results of alternative assessment approaches can be used 
to ( l ) determine the extent to which a project is attaining its goals; (2) 
ascertain changes that will improve the overall effectiveness of a 
project; and (3) identify successful components of a project. 

Staff can strengthen the use of alternative approaches by ( I ) 
collecting data systematically; (2) analyzing the data to provide 
meaningful interpretations of the results of assessments; (3) using the 
data to answer questions that staff, learners, and others have about the 
project; and (4) creating informative reports that summarize the results 
of the assessments. 
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By considering 
authentic representations of 
what learners and staff are 
doing, alternative assessments 
can provide clear insights into 
the effects of the project on 
adult learners and their 
children. 



Surveys, performance samples, interviews, and observation 
measures are four kinds of alternative assessments which, together 
with standardized assessment measures, can be used to create a 
comprehensive evaluation design for family literacy projects. To 
maximize their effectiveness, alternative assessments need to be 
tailored to the unique goals, objectives, and program design of the 
project. By considering authentic representations of what learners and 
staff are doing, alternative assessments can provide clear insights into 
the effects of the project on adult learners and their children. As an 
integral part of identifying needs, assessing ongoing progress, and 
determining project effectiveness, alternative approaches play an 
indispensable role in enabling staff and learners to collaborate in 
designing a project that is responsive to the needs of language- 
minority learners, families, and communities. 
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CHAPTER VI 

NEXT STEPS: USING 
THE RESULTS 

TO REFINE 
THE PROJECT 



Judith B. Wilde 
Evaluation Assistance Center, Western Region 
New Mexico Highlands University 
Albuquerque, New Mexico 



Chapters I through V of this handbook describe processes and procedures to 
be followed from the early development of a Family literacy project through 
the collection and reporting of data to determine how well the project is 
progressing. Throughout the volume the authors have emphasized the 
importance of linking assessment and evaluation to the needs of the learners. 
This chapter will describe how the authors attempted to link the contents of 
the handbook to the needs oF staff members in family literacy projects. * The 
chapter will demonstrate how the planning process model introduced in 
Chapter II was used to guide the development of this handbook. 

This chapter is divided into three sections: (I) overview of the development 
of the handbook; (2) description of the publication of the revisions to the 
handbook; and (3) additions to the handbook according to feedback from 
reviewers. 

*?>ec the Glossary on pages 137-1 38 of this publication Tor definitions of italicized key terms thai 
appear in this chapter. 
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The project directors 
emphasized that family literacy 
projects are extremely diverse 
in design, curricula, 
populations served, languages 
served, purpose, and location. 



Use of ihe Planning Process Model 
io Develop ihe Handbook 

The handbook was first developed in 1991 and field- tested in 
1992 and 1993. This section describes the process used to develop and 
Held test the volume. The process was very similar to the planning 
process model described in detail in Chapter II. The model is used in 
this section to help the reader understand the rationale for the design 
of the handbook and to illustrate how a resource document can be 
developed and then revised collaboratively with the users of the 
volume. 

Development of the Field-Test Edition 

In December 1990, the California Department of Education's 
Bilingual Education Office (BEO) held a one-day seminar for project 
directors of the state's Family English Literacy Programs (FELP). 
The seminar was funded by the Office of Bilingual Education and 
Minority Languages Affairs (OBEMLA) through Title VII of the 
Elementary and Secondary Education Act (ESEA) — also known as 
the Bilingual Education Act. At the seminar, the participants identi- 
fied several issues of importance: 

• Standardized tests are rarely available and rarely appropriate for 
an adult population of English learners. 

• Alternative assessments, though appropriate, arc not readily 
available at present. 

• Few guidelines have been published describing the development 
of alternative assessments for this population. 

The project directors also emphasized that family literacy 
projects are extremely diverse in design, curricula, populations 
served, languages served, purpose, and location (for example, school, 
apartment, or church). Thus, an instrument created for one project 
would probably be inappropriate for another without major revisions. 
The participants discussed ways to meet their assessment and evalua- 
tion needs and identified ways in which BEO and OBEMLA could 
provide assistance. The most effective approach seemed to be the 
development of a handbook on assessment. The seminar and immedi- 
ate follow-up to it represent the three steps of the preplanning phase of 
the planning process model. Table 1 illustrates the first three steps of 
the model and its role in the development of the handbook. 
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Planning Process Model: Preplanning Phase 
of Handbook Development 




- ~=*Doveippm0ntQl Activities 



1 Develop goals 



BEO plans FELP seminar 



2 



Conduct needs assessment 



FELP directors identify issues 



3 



Design objectives 



BEO develops idea of 
handbook, identifies authors 



Two themes were 
repeated throughout the 
discussion— the need for 
alternative assessments at ail 
phases of a family literacy 
project and the need to help 
project directors improve their 
projects. 



BEO staff held a second meeting of FELP directors in January 
1 99 1 . This meeting took the form of a focus group, led by two of the 
authors. The directors and invited guests discussed three specific 
aspects of a family literacy project: 

1 . Intake — strategies and formats for assessing learners' levels of 
literacy and related abilities. 

2. Instruction — strategies and formats for assessing learners' 
progress in meeting the objectives of the project. 

3. Evaluation — strategies and formats for assessing the overall 
goals of the project. 

Two themes were repeated throughout the discussion — the need 
for alternative assessments at all phases of a family literacy project 
and the need to help project directors improve their projects. 

After the meeting, the outline of the handbook was discussed 
with OBEMLA personnel and other agencies that might provide 
assistance. The authors held several subsequent meetings, identified 
other educators who might participate in the project, and wrote and 
edited the chapters. Time lines were established for drafting, revising, 
and publishing the handbook. The field-test version was published in 
September 1992, with the assistance of the FELP project at Miami* s 
Florida International University. The authors discussed a time line for 
revisions and the need for a field- test document. This portion of the 
development of the handbook parallels the planning phase of the 
model shown in Table 2. 
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Planning Process Model: Planning Phase 
of Handbook Development 



The usage surveys 
provided both quantitative and 
qualitative data. 



-^sps In the Planning 

4 Develop curriculum 

and instructional activities 

5 Identify obstacies 



6 Identify necessary resources 

7 Assign responsibility 

8 Establish time line 



Focus group: FELP project 
directors, authors, BEO staff 

BEO staff discuss outline of 
handbook with OBEMLA 

Meetings with authors 

Writing and editing of chapters 

Drafts, revisions, publishing 



Field-Testing the Handbook 

Before the field-testing of the handbook during the 1 992-93 
academic year, the authors discussed various methods of soliciting 
comments. Although each option presented advantages and disadvan- 
tages, the best option seemed to be a series of short rating scales, one 
included at the end of each chapter of the volume. Each rating scale or 
us".ge survey could be completed as the chapters were read. The usage 
surveys listed the major topics of each chapter and asked the reader to 
rate the clarity of the writing and the information provided. In addi- 
tion, readers were given the option of making other comments and 
suggestions. This provision offered the possibility of obtaining both 
quantitative (numeric) and qualitative (reflection) data. The usage 
survey for Chapter I is shown in Figure 1 on page 134. Table 3 
illustrates the field-testing of the handbook and planning for revisions 
during the evaluation phase of the model. 



Planning Process Model: Evaluation Phase 
of Handbook Development 



Steps in the Planning 
Process Model 



-Dey&Jqptngntal Activities 



9 Specify outcomes 

10 Design instruments and 
collect data 

1 1 Analyze data 



12 Formulate conclusions 



Decide to revise handbook 
Create usage surveys 

Analyze qualitative and quantitative 
data from usage surveys 

Identify publisher, revise handbook 
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In general, reviewers 
rated the handbook as quite 
beneficial and especially liked 
the sample assessment 
instruments. 



The authors were very 
interested in having the 
handbook edited and produced 
by individuals who were 
knowledgeable about lamiiy 
literacy and wanted the 
handbook disseminated widely. 



Approximately 30Q copies of the handbook were disseminated: 

• To all FELP projects in the United Slates that began operations 
between 1990 and 1992. 

• To coordinators of bilingual education programs in California. 

• To offices of bilingua] education in each of the 50 state departments 
of education. 

• To selected experts in adult literacy, bilingual education, and family 
literacy. 

In addition, presentations on the handbook were made at various 
national, regional, and state meetings and conferences, such as the 
national conference of the National Association for Bilingual Educa- 
tion; a national meeting of Family English Literacy programs funded by 
ESEA Title VII; a meeting of Even Start projects in California; and 
various meetings of educators interested in the evaluation of language- 
minority programs. 

Results of the surveys. Most of the comments received about the 
handbook were positive. Some individuals returned the usage surveys, 
and others sent letters or telephoned with information of a more qualita- 
tive nature. In an attempt to obtain focused* critical remarks, a selected 
group of recipients — primarily FELP directors and experts in adult 
literacy — were sent letters requesting feedback. In general, reviewers 
rated (he handbook as quite beneficial and especially liked the sample 
assessment instruments. Others stated that the document would be very 
useful during the planning stages of a family literacy project. 

Some reviewers provided useful criticisms of the handbook. For 
example, some requested more information about standardized tests and 
workplace literacy; others thought that the handbook should emphasize 
strongly that the results of evaluation should be used for improving the 
project. All the feedback received during the field- test was considered 
by the authors during the revision process. This chapter is itself an 
outcome of the field-test, particularly the sections on standardized tests 
and workplace literacy. 

Publication of the Handbook 

The authors were very interested in having the handbook edited 
and produced by individuals who were knowledgeable about family 
literacy and wanted the handbook disseminated widely. For these 
reasons they selected the National Clearinghouse for ESL Literacy 
Education (NCLE) at the Center for Applied Linguistics in Washington, 
DC, as the publisher. 

The process of selecting the publisher provides further illustration 
of the planning process model. In determining the type of publisher best 
able to disseminate the handbook, the authors had to reexamine the 
goals of the handbook. Submitting the manuscript to NCLE constituted 
a new needs assessment as the authors responded to feedback from 
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NCLE staff. Table 4 illustrates how completing one planning.cycle 
brought the authors back to the preplanning phase as they Preplanned" 
the handbook. 



Planning Process Model: The On-going 
Process (Revisions) 





— . — ■ ■ ■ — . . - 
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Preplanning: Develop goals Authors plan revisions 



Preplanning; Conduct needs 
assessment 

Preplanning: Design 
objectives 



4-B Planning 



9-12 Evaluation 



Find publisher, send Field-Test 
manuscript 

Revisions from NCLE and 
usage survey respondents 

Authors revise manuscript; add 
Chapter VI 

Comments from NCLE staff 
and BEO editors 



The assessments, 
procedures, evaluation 
designs, and reporting 
methods presented in the 
handbook will be appropriate 
for literacy projects supported 
by many funding agencies. 



Revisions to the Handbook Manuscript 

Most of the revisions in the handbook were related to expanding 
the focus of the volume from the Family English Literacy Program 
funded by ESEA Title VII to family literacy projects in general. 
Although the volume still concentrates on programs serving language- 
minority adults and families, many of the suggestions could also be 
used in literacy programs for monolingual English speakers. Rather 
than concentrating on the regulations for one funding source (ESEA 
Title VII), the handbook now reminds readers that they need to 
address the concerns of the if particular funding agency. The assess- 
ments, procedures, evaluation designs, and reporting methods pre- 
sented in the handbook will be appropriate for literacy projects 
supported by many funding agencies. 

Some reviewers suggested that the authors provide more infor- 
mation about standardized tests and workplace literacy. Because the 
focus of the volume is alternative assessment and evaluation, the 
authors did not expand their chapters to include those topics. How- 
ever, they did believe thai readers ought to have more information 
about the ways in which standardized assessments could complement 
alternative approaches in an overall evaluation design for a literacy 
project. They also thought it important to provide readers with sources 
of more information about workplace literacy. The next section of this 
chapter deals with standardized tests and workplace literacy. 
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Although standardized 
tests probably are not the best 
measure for adult learners, 
they do provide certain * ypes 
of information that instructors 
might find helpful if interpreted 
appropriately. 



According to Figueroa, 
'tests in English degenerate in 
unknown degrees into tests of 
English language proficiency." 



Additions to the Handbook 

This section deals with two topics for which readers of the field- 
test version requested information — standardized tests and workplace 
literacy. Although they are not major issues in the volume* they are 
related to the implementation of many family literacy projects. For 
instance, funding sources usually require a measure of learner progress, 
implying that the measure should be a standardized test. Although 
standardized tests probably are not the best measure for adult learners, 
they do provide certain types of information that instructors might find 
helpful if interpreted appropriately. Workplace literacy, on the other 
hand, is a newer field of interest. It is quite possible that those working 
with adult learners might want to determine what aspects of workplace 
literacy they might be able to include within their programs and where 
they can find further information on the topic. This section is designed 
to meet those very real needs and concerns- 
Standardized Tests 

A standardized test of academic achievement, language profi- 
ciency, or affective traits is typically used to determine learners* skills, 
proficiencies, or strengths within a particular domain. The tests are so 
named because the administration, format, contents, language, and 
scoring procedures are the same for all students; that is, they have been 
standardized. Because of this, all learners theoretically have the same 
opportunity to demonstrate their level or amount of achievement, 
proficiency, or affective trait in comparison to a large group of other 
learners. This second group of learners (norming group) was selected by 
the test publisher to take the test; they are usually described in the test's 
technical manual. The norming group should include some individuals 
who are similar to the students taking the test; for example, they should 
be similar to the family literacy project's learners in gender, age, race or 
ethnicity, cultural background, language background. English profi- 
ciency, and educational background. Because of these similarities, it is 
appropriate to compare the results of the two groups. However, prob- 
lems with standardized tests should be considered before the results 
are used. 

Limitations of standardized tests. Standardized achievement tests 
have been attacked since 1934 for being biased against many groups of 
students. The attacks cite cultural, linguistic, and ethnic bias (sec 
DcAvila & Havassy. 1974; Figueroa, 1990; Sanchez, 1934) as well as 
gender bias (Loewcn. Rosser, & Katzman, 1992). Because most stan- 
dardized achievement tests are administered in English* this problem 
has special meaning for language-minority students. Figueroa ( 1990) 
states that "tests in English degenerate in unknown degrees into tests of 
English language proficiency' 4 (p. 93). Even though some tests have 
been translated into other languages, most notably Spanish, DcAvila 
and Havassy argue that these tests 41 as they arc currently designed are of 
little use to anybody 14 ( 1974* p. 72). 
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Much of the problem 
with proficiency tests is 
associated with the lack of a 
generally accepted definition of 
proficiency. 



The principal strengths 
of standardized tests are their 
ready availability and capacity 
to provide comparisons among 
learners and across time 
periods. 



Standardized tests of language proficiency present d different set 
of problems. First, most proficiency tests being used place a great deal 
of emphasis on discrete skills, such as grammar and vocabulary. They 
tend not to measure the more complex constructs (Bachman, 1990; 
Duran, 1988; Harley, Allen, Cummins, & Swain, 1990). Much of the 
problem is associated with the lack of a generally aceepted definition 
of proficiency. For example, most proficiency tests still measure only 
oral production skills — just one pan of what is needed for communi- 
cating with different people in different settings (Durdn. ] 988). 

Strengths of standardized tests. Standardized tests are commonly 
used in programs for language- minority learners because they are 
readily available and relatively inexpensive. In addition, they can be 
scored by the test publisher (thus releasing teachers from an onerous 
job) and provide for comparisons among groups. Moreover, they allow 
for comparisons to be made from one time period lo another — if 
normal curve-equivalent (NCE) scores or scale scores are used. 

Berlak explains another reason that standardized measures are 
commonly used: "Since the turn of the century in North America, 
psychometricians have laid claim to providing the most, if not the only, 
scientifically valid means of measuring academic achievement" 
(original emphasis, 1992. p. 194). The validity and reliability of :;iost 
standardized tests can be documented readily; many studies have been 
done by test publishers and others lhat support the technical merits of 
the tests. However, the studies generally include only middle-class 
learners and rarely involve language-minority students or older stu- 
dents. Thus, the principal strengths of standardized tests are their ready 
availability and capacity to provide comparisons among learners and 
across time periods. 

Appropriate uses of standardized tests. The following are some 
caveats to keep in mind when considering the use of standardized tests: 



I. 



3. 



4. 



Select a test thai measures at least 40% of the content taught in 
the program — a condition that can be especially difficult for a 
family literacy project. 

Select a lest that is sensitive to the cultural, linguistic, and age 
differences of the learners. 

Interpret test scores carefully. Because no test is perfect, each 
score is only an estimate of the learner's true ability. 

Never use one cut-off score. Create a range or band- width of 
scores within which further assessment is required. For example, 
a score below 40% correct indicates lhat the student needs further 
learning experiences; 40 to 50%, more assessments are necessary 
to judge the learner's achievement; and over 50%, the learner can 
move to a new learning area. 

Examine carefully the background information about the test to 
determine the test's relevance for the project (for example, the 
composition of the norming group and the contents of the lest). 



2 



Assessing Success 



136 



BEST COPY AVAILABLE 



Alternative assessments 
take place in a contextuatized, 
real-life totting, thereby 
providing a more performance- 
based measure of the learner's 
abilities. 



6. Compare the test results with the learner's performance in the 
classroom to ensure that learners with higher scores on the test 
are those who perform better in class. 

7. Be sure to request scale scores or NCEs. 

8. Consider the results of supplemental assessments and oihcr 
information about the learner so that decisions are not based on 
one standardized test score (Evaluation Assistance Center-West, 
1992). 

The focus of most of the handbook has been on alternative 
measures. Such assessments are more appropriate for many learners, 
especially those in family literacy programs. Alternative assessments 
take place in a contextualized, real-life setting, thereby providing a 
performance -based measure of the learner's abilities. Alternative 
approaches usually afford learners with a greater opportunity to 
demonstrate their actual abilities. Standardized testing, on the other 
hand, lakes place in a dec on textual ized, neutral setting, which can 
work against learners whose test-taking skills are not as well devel- 
oped. (See Navarrete et a!., in press, for a thorough discussion of 
dec ontextual ized versus contextualized measures,) Standardised tests 
can, however, be modified to be perfor. jance based. Table 5 provides 
some example test items from stand ard ized multiple-choice tests 
together with suggestions on how they might be modified to yield 
meaningful results. 



Sample Test Items: Multiple Choice 
and Performance Based 



Standardised item . 



TFQrfQFwance based item 



What is 3 + 4 ? 



Find the area of the shape below 
(a 3 x 5 rectangle is pictured). 

Look at the table showing how 
many cars of different makes 
were sold las! year. How many 
Toyola Celicas were sold? 

Round 23 to the nearest 1 0, 

Add 56 and 68. 



The answer is 7. What is the 
question? or Draw a picture that 
shows 3 + 4, 

Draw a shape with an area of 1 6 
square units. 

Look at the table below. What does 
it tell us about the cars that were 
sold last year? 



What numbers round to 20? 

Tucker adds 57 and 68 on his 
calculator and gets 113. How do 
you know it's wrong? 



Note: Modified from Evaluation Assistance Center-West, 1993. 
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^ To find a standardized 
test: 

• Review the objectives of the 
project. 

• Determine the viability of a 
few tests. 

• Use one test on a group of 
learners. 

• Provide the learners with test 
faking skills. 



Finding an appropriate standardized test. By following the 
caveats described in the previous section, project staff may be able to 
identify a standardized test that meets the needs of their project. The 
following are suggested steps for selecting an appropriate standard- 
ized lest: 

1 . Review the objectives of the project. If necessary, put them in 
hierarchical order with the most important objective first and the 
least important last. 

2. Identify four or five potential tests by contacting other family 
literacy projects, reading evaluation reports or research studies to 
determine what measures were utilized, contacting technical 
support agencies, and looking for reviews of tests in professional 
journals or magazines. 

3. Determine the viability of the tests by consulting professional 
test review sources (for example, The Ninth Mental Measure- 
ments Yearbook, edited by Mitchell, 1985, or Test Critiques, by 
Keyser & Sweetland, i984), to ensure that the test meets the 
needs of the project and has the appropriate characteristics of 
reliability and validity. The possibilities should thus be narrowed 
to two or three tests. 

4. Request the technical manual and a review copy of each of the 
possible tests. Determine whether there is a match between the 
test objectives and the project objectives and a match between 
the demographics of the norming group for the test and the 
learners in the project. One test should now stand out as the best 
option, 

5. Use the test on a group of learners — ideally, learners from two 
very different skill levels — to ensure that they feel comfortable 
with the test, that the test can differentiate between learners with 
diftercnt skill levels, and that the scores provide useful informa- 
tion about the learners. 

6. If a decision is made to use the test on a regular basis, provide 
the learners with some training in test-taking skills. This sugges- 
tion does not mean teaching to the test but rather helping learners 
develop such strategies as making good choices among alterna- 
tive responses, using time wisely, and skipping difficult items. 

Some of the standardized tests that are typically used in family 
literacy projects are listed in Table 6. Also provided in the table are 
brief descriptions of the purpose of the test and the language require- 
ments for test takers. Several books and articles are available that 
provide more information about such tests (see Jackson, 1990). In 
addition, some tests of proficiency are available and appropriate for 
adult students: for example, Adult Language Assessment Scales (A- 
LAS) and the IDEA Profic iency Test (fPT) (Dalton, 1989, 1992; De 
Avila & Duncan, 1993). 
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Table 6 



Standardized Tests Commonly Used in 
Family Literacy Projects 



Workplace literacy 
projects are designed to 
provide learners with work- 
related literacy skills, including 
the communication skills 
necessary to ask questions, 
clarify concerns, and solve 
problems cooperatively through 
discussion with others. 



;• » 

- - - Tact 

z = W§f. 


nksL&jfvj*** feimw- r 


Literacy 1 


Adult Basic Learning 
Exam (ABLE) 


Reading, writing, 
math 


Grade 2 English 


Basic English Skills Test 
(BEbT) 


Speaking, listening, 
reading, writing 


Low proficiency, 
nonnative English 


CASAS Adult Life Skills 


Reading 


Grade 1 English 


Adult Language 
Assessment Scales 
(A-LAS) 


Reading, writing, 
math, s pea kisg, 
listening 

r 


Grade 2 English 


Reading Evaluation Adult 
Diagnosis (READ) 


Reading 


Grade 1 English 


Spanish Assessment of 
Basic Skills (SABE) 


Reading, math 


Grade 1 Spanish 


Tests of Adult Basic 
Education (TABE) 


Reading, writing, 
math 

f 


Grade 2 English 


Tests of Adult Basic 
Education (TABE) — 
Spanish 


P relite racy/ 
literacy skills 


Grade 1 Spanish, 
proliferate Spanish 



In summary, as suggested by Navarrele el al. (in press), standard- 
ized tests of basic skills and language proficiency can i>e used if the 
following conditions are met; 

1 . They are not the sole measurement of learners' achievements. 

2. Scores arc interpreted carefully. 

3. They match the demographic and educational backgrounds of the 
learners in the project. 

4. The objectives of the lesl match the objeclivcs of the prujeci, 
including the curriculum content. 

Workplace Literacy 

The purpose of many family literacy projects is to address the 
literacy needs of adults and children in families in which a language 
other than English is spoken. The projects have several components, 
some of which might include native-language literacy, English 
literacy, cross-cultural development, self-esteem, family learning, and 
home-school relations. Workplace literacy projects are designed to 
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Workplace literacy 
programs are essential to the 
economic and social growth of 
all learners and employees. 



Individuals planning 
workplace literacy programs 
need to consult a variety of 
sources to obtain a 
comprehensive understanding 
of learners' work-related 
needs. 



provide learners with work-related literacy skills, including the 
communication skills necessary to ask questions, clarify concerns, 
and solve problems cooperatively through discussion with others 
(Council of Chief Stale School Officers, 1991 ). Workplace literacy 
projects may be considered a type of family literacy project, but their 
goals are more specifically work related. 

Current research has identified four types of workplace literacy 
projects characterized by the following attributes (Gowen, 1 992; 
Kutner, Sherman, Webb, & Fisher. 1991; Wiigley & Guth, 1992): 

1 . Programs to give prospective workers the language skills they 
need to enter the work force. 

2. Short-term courses in e idler English or the native language to 
prepare learners for — or improve their performance on — a 
specific set of job-related language tasks. 

3. Programs that focus the curriculum design on the needs and 
goals of the learners as well as diose of the sponsoring agency. 

4. Programs for professionals who have at least basic literacy skills 
in their native language and who need to improve their English- 
language abilities. 

Workplace literacy programs are essential to the economic and 
social growth of all learners and employees, especially those whose 
English-language proficiency is not sufficient to allow them to seek 
employment successfully. Bilingual programs are a growing trend in 
both short-term courses and the learner-centered programs as effec- 
tive, efficient approaches to meeting learners' needs. Although this 
volume does not address workplace literacy, sources of information 
are available, McGroarty (1993, pp. 102-4) has identified four docu- 
ments that can be suggested for those who wish to increase their 
knowledge of workplace literacy: 

• Cichon, Graver, and Thomas ( 1991 ) provide a comprehensive 
overview of the types of programs available for teaching 
language -minority workers. 

• Thomas, Bird, and Grover (1992) provide a monograph aimed 
specifically at community college educators interested in offering 
vocational training to language -minority learners, 

• Thomas, Grover, Cichon, Bird, and Harns v 1991 ) provide a step- 
by-step guide for program planners considering the establishment 
of workplace literacy programs. 

• Wrigley and Guth (1992) provide a handbook for researchers, 
program developers, and policymakers that contains definitions of 
literacy and describes seven approaches to designing a curriculum 
for adult literacy programs. 

This brief overview indicates that individuals planning work- 
place literacy programs need to consult a variety of sources to obtain a 
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The planning mode! is 
important in the development 
of curriculum, grant proposals, 
business plans, and other 
activities that require effective 
planning and evaluation. 




comprehensive understanding of learners' work-related needs. Read- 
ers are encouraged to review the reference section to Obtain more in- 
depth information and assistance. 

Summary 

It is difficult for one handbook to meet the assessment and 
evaluation needs of all participants in a wide variety of family literacy 
projects. This volume has focused on designing and using alternative 
approaches to assessment and evaluation in order to improve literacy 
projects. By casting the development of the handbook itself within the 
schema of the planning process model introduced in Chapter II, this 
chapter has described the history of the volume and illustrated how a 
needs-based project can be developed. The planning model is impor- 
tant not only within the context of developing a specific type of 
literacy project but also in the development of curriculum, grant 
proposals, business plans, and other activities that require effective 
planning and evaluation. By illustrating how the model was used to 
develop this handbook, the authors hope that staff members in family 
literacy projects will find uses for the model and the handbook to 
improve their projects. 
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Sample Survey of Usage from Field-Test Edition 



Chapter I. Introduction to Alternative Approaches to Assessment and Evaluation 

This resource guide must be helpful to you, the FELP director or evaluator, in order to be considered a 
success. In accepting the field- test edition of the guide, we hope thai you agree to participate in this usage 
survey. We plan to revise the guide based on your comments and criticisms; it then will be published for a 
broader audience. Please share wilh us your critical comments. After reviewing the chapter, use the scale 
below to indicate whether the guide is written clearly; the scale indicates that it is 

1. Clearly written and understandable; 

2. Generally clear, but some wordings or format could be improved; 

3. Not particularly clear, written with too much detail or too much jargon; or 

4. Not at all clear, not at all understandable. 

In addition to being written clearly, the guide also must provide enough information for you to be able 
to implement the ideas. In each of the same areas, please use the scale to indicate that the guide has 

1 . Valuable information in the right amount of detail for me really to use it; 

2. Good information, but needs more detail for me really to use it; 

3. Information that is tangential to my needs, so it won't help a great deal: or 

4. Information that I don't need and won't use. 

Rate clarity Area to be rated Rate information 

1 2 3 4 Format of the chapter Not applicable 

1 2 3 4 Wording of the chapter Not applicable 

Not applicable The general ideas and information presented 12 3 4 

1 2 3 4 Types of assessments (e.g., alternative, standardized) 12 3 4 

1 2 3 4 Role of evaluation 12 3 4 

1 2 3 4 FELP statute and explanations 12 3 4 

1 2 3 4 Evaluation issues for FELP projects 12 3 4 

1 2 3 4 ESEA Title VII evaluation requirements 12 3 4 

1 2 3 4 Overview of the guide 12 3 4 

Thank you for your assistance. If there are olher comments, criticisms, or assistance you would HVe to 
i,ffer, please feel free to do so. You do not need to give your name. 



Please return your completed surxey. You may fold, staple, and stamp this page — the address is on the 
back side. If you prefer, you may FAX it to Daniel D. Holt, Bilingual Education Office at (916) 657-2928. 
We also would appreciate receiving a copy of any alternative assessments you create as a result of this 
guide. 
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GLOSSARY 



Alternative assessment. A flexible assess- 
ment approach that is representative of the 
curriculum, meaningful to learners, and 
indicative of the learners' acquired ability or 
knowledge. Examples are surveys, interviews, 
observation measures, and performance samples. 

Annual evaluation report. Submitted yearly to 
a funding agency such as the Office of Bilingual 
Education and Minority Languages Affairs 
(OBEMLA), U.S. Department of Education, to 
summarize the accomplishments of a project. 

Assessment. The use of instruments and 
procedures to gather data on a regular basis. 
Assessment may focus on identifying learners" 
needs, documenting the learners' progress 
toward meeting their own goals, and ascertaining 
the extent to which the project objectives are 
being met. 

Baseline data. Data that are collected before 
project activities are implemented. For example, 
baseline data collected on new enrol Ices may be 
used for comparison with data obtained from 
learners at a later point in the project. 

Descriptor. An explanation of an individual 
rating on a scale. For example, on a 0-5 scale for 
assessing reading, a descriptor for 0 may be 
Cannot read instructions and the descriptor for 
5 might be Reads instructions with ease. 

English as a second language. English 
acquired by individuals whose native language is 
a language other than English. 

Evaluation. The process of integrating and 
analyzing assessment data at a given point in 
time for such purposes as developing and 
refining goals and objectives; documenting the 
learners' progress; and determining the overall 
success of the project. 

Family English Literacy Program (FELP). 

A program authorized in 1984 as part of Title 
VII of the Elementary and Secondary Education 
Act. FELP is defined as "a program of 
instruction designed to help limited-English- 



proficient/LEP adults and out-of-school youth 
achieve competence in the English language. . . . 
Such programs may include instruction on how 
parents and family members can facilitate the 
educational achievement of limited -English- 
proficient children." [20 USC §3283(a)(7)J 

Family literacy project. Consists of a set of 
inter-related components that address the 
literacy needs of adults and children. This 
handbook focuses on families in which a 
language other than English is spoken. Examples 
of components are native language literacy, 
English literacy, cross-cultural development, 
self-esteem, family learning, and home-school 
relations. 

Final performance report. Submitted to a 
funding agency such as the Office of Bilingual 
Education and Minority Languages Affairs 
(OBEMLA), U.S. Department of Education, to 
summarize the accomplishments of a project 
during the entire grant period. 

Focus group. A group that is formed for the 
purpose of providing input. Group members are 
interviewed according to a protocol that has 
been previously established. 

Frequency count. A technique for tallying 
information, usually from survey responses. A 
frequency count may provide information such 
as how many parents need child care or how 
often participants visit the library. 

Holistic scoring. A strategy for assessing 

performance by summarizing individual features 
to obtain an overall view of learners" progress or 
some other dimension of the project. 

Instructional Program. A coordinated set of 
acti viues that represent the curriculum and 
instructional components of a project. 

Interview. An alternative assessment approach 
for obtaining detailed information. An interview 
may be administered to individuals or to groups 
with respondents answering orally or in writing. 
Interviews may be used with focus groups. 
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Literacy. The ability to use language and 
language-related strategics, especially itose 
associated with reading and writing, in a variety 
of settings (family, community, school, *ork) to 
achieve one's own goals (personal, profes iional, 
academic). 

Literacy framework. Staff members' and 
learners' shared understanding of the learners' 
literacy needs, program goals, curriculum and 
methodology, and approaches for assessing and 
evaluating literacy development. The elements of 
the framework should be coordinated so that they 
are interdependent and mutually supportive. 

Literacy practices. The ways individuals use 
literacy in their daily lives. 

Modified cohort design. A method of 

analyzing data by comparing the performance of 
project participants with comparable individuals 
who have not yet benefited from project services. 

Norm-referenced assessment. Sec 

Standardized assessment. 

Objective. A description of an outcome or change 
that is expected as a result of implementing a 
project. The content of objectives is determined 
by what the learners need, what the staff are 
prepared to leach, and what is possible to achieve 
within the time and budget constraints of the 
project. 

Observation measure. An alternative 
assessment instrument for obtaining various 
kinds of information. Observations may focus on 
such features as learners" characteristics, group 
interactions, or language and literacy 
performance. 

Performance sample. An example of a 
learner's work in selected tasks. Performance 
samples include writing samples, oral readings, 
role plays, and oral presentations* 

Portfolio. A collection of examples of a learner's 
accomplishments such as reading logs, language 
experience stories, and writing samples. The 
contents of the portfolio can be used to show a 
learner's progress over time. 
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Project. Refers to all components to be 

implemented, including those described in the 
instructional program. 

Protocol. A specified set of procedures and 
questions designed to achieve the purposes of an 
assessment activity. 

Rating scale. A technique for assessing an area 
of interest (for example, reading ability or self- 
confidence) on a scale of intervals usually 
designated by numbers or descriptors. 

Reliability. Refers to the capacity of an 

assessment instrument or procedure to produce 
approximately the same results consistently on 
different occasions when the conditions of the 
assessment remain the same. For example, will 
the instrument provide the same results if 
learners are tested on Thursday instead of 
Wednesday, or if two or more competent judges 
scored the test independently? 

Standardized assessment. An instrument 
that is created to allow for comparisons bciween 
individuals' current achievement and the 
average performance (norms) of selected 
participants (norming group). An example is the 
Basic English Skilb Test (BEST), available from 
the Center for Applied Linguistics, Washington, 
DC. In this volume, the terms standardized 
assessment and norm-referenced assessment are 
used interchangeably. 

Survey. An alternative assessment approach for 
obtaining information. A survey may be 
administered to individuals or groups with 
respondents answering orally or in writing. 

Validity. Refers to the capacity of an assessment 
instrument or procedure to measure what it 
claims to measure. For example, does the 
reading test actually measure reading 
comprehension, or does success on the test also 
depend on the learners* knowledge of U.S. 
culture? 
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accountability, 27 

activities 

collaborative, 70-71 
instructional, 26, 72 
project, 30, 68 

Adult Language Assessment 
Scales (ALAS), 129 

affective factors, 72 

assessment, 1, 6 
closed-ended, 3, 37 
collaborative, 17 
conventional, 62 
framework, 19 
initial, 1 1,35-37 
needs (see needs assessment) 
open-ended, 3, 38, 82 
progress, 61-62 
project-based, 19 
self, 80-S4 

standardized, 1, 2, 33, 125-129 
strategies, 29, 65, 68-76 

assessment, alternative 
design, 30 

instruments, 3, 17, 29 
interviews, 3. 21, 37, 39, 104- 
108 

investigations, 3 
journals, 3 
learner-centered, 64 
observation measures, 3, 21, 

73,87-88, 108-117 
performance samples, 3, 21, 

97-103 
portfolios, 3,4,45,76, 100 
procedures, 2, 29, 37 
surveys, 3, 37-39, 94-97 

attendance, 8, 1 1 



B 

baseline data, 8, 10, 25, 45, 65, 
68 

Basic English Skills Test (BEST), 
2, 129 

biliteracy, 8 

c 

Cambodians, 38, 40 
(see also Khmer) 

Chinese, 28 

citizenship, 38 

cognition, 72 

collaboration, 53 

community liaisons, 30 

comparison group, 10 

computers, 28 

Comprehensive Adult Student 
Assessment System (CASAS), 
2, 129 

cultural 

adaptation, 1 1 
background, 37 
cross-cultural sensitivity, 23, 
65 

cross-cultural strategies, 43 

curriculum, 62 
content, 20 
development, 26 
learner-centered, 63 
literacy, 20 
ready-made, 19 

D 

data 

analyses, 29 

baseline, 8, 10,25,45, 65,68 
collection, 32 
demographic, 37, 39 
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focus group, 104-108 
frequency count, 4, 29, 95, 100 
implementation, 10 
interpretation, 106 
longitudinal, 1 13 
observation, 1 10 
outcome, 10 
raw, 106, 108 
survey, 94-97 

demographics, 37, 39 

development 
cognitive, 20 
linguistic, 19 

diversity, 6 

discrete skills, 2, 17 

E 

Elementary and Secondary 
Education Act (ESEA), 1, 10 

enrollment, 8, 26-27 

evaluation, 1,6, 10, 32, 122 
design, 6, 10 

modified cohort, 108, 1 12-1 13 
process, 20 
product, 1, 32 

evaluator, project, 30, 32 

F 

Family English Literacy 

Programs (FELP), 1,7, 19,24- 
28, 123 

focus groups, 3, 37, 39, 104- 108, 
109, 122 

framework 

literacy, 9, 17, 19-20, 23,40 

Freire, Paulo, 31,72 

frequency count, 4, 29, 94, 100 

funding sources, 10,40 
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G 

goals {see also objectives) 
statements, 23 

grammar, 62 

grant proposals, 40 

H 

Hmong, 28 

I 

IDEA Proficiency Test (IPT), 128 

instruments, 4 

self-assessment, 80-84 
standardized, 2 

intake 

assessment, 1 1, 43-45, 55 

intelligence, 16-17 

interviews, 3, 21, 37, 39, 43-44, 
104-109 

K 

Khmer, 28, 38 
Korean, 28 

L 

language (see also native 
language) 
abilities, 69 
development, 1 1 
groups, 37 

productive skills, 1 10 
receptive skilly 1 10 

language minority students, 1, 18, 
125 

Language Experience Approach 
(LEA), 73-74 

Lao, 28 

learner (see also participants) 
behavior, 66 
cultural background, 8 
needs, 4 



profiles, 66 
progress, 4 

learner centered 
approach, 63 
assessment, 64 
curriculum, 62 
programs, 19 

learning 

views of, 16-17 
strategies, 19 

literacy 

activities, 18, 72, 76 
awareness, 5, 25 
brokers, 65 
contexts, 18 
curriculum, 20 
definition, 19-20 
development, 18, 76 
emerging, 76 
experiences, 41 
framework, 9, 17-20,23,40 
inventories, 63 
paradigm, 4 1 
practices, 18-19, 64,68 
quantitative, 18 
workplace, 129-131 

longitudinal 

analysis, 112-113 
data, 113 
progress, 104 

M 

Mental Measurements Yearbook, 
128 

modified cohort design, 108, 
112-113 

N 

National Council of La Raza 
FELP project, 25 

native language, 8-9, 37, 41, 43, 
44,65, 104 
abilities, 5 
literacy, 41 
proficiency, 41 



needs assessment, 6, 24-25, 36- 
37,40, 48-51 

normal curve equivalent scores, 
L26 

norming group, 2 
norms, 2 

0 

objectives, 5, 29 
child, 9 

instructional, 5, 9, 17, 25, 36 
parent, 9 
project, 33 

observation 
card, 66 

measures, 3, 21.73, 87-88, 
108-117 

P 

parent-child interactions, 68 

parent survey, 42, 50-5 1 

Painting Curricuiuni for 
Language Minority Parents, 
28 

participants 
enrollment, 27 
recruitment, 26-27 

performance samples, 3, 21, 97- 
103 

phonics, 62, 73, 76 

planning, 30-3 I 
collaborative, 19 

planning process model, 21-22, 
120-122, 131 

portfolios, 3,4,45, 71,75, 100 

Proceedings from the Family 
English Literacy Seminar, 8 

program design, 7 

progress 

assessment, 6 1-62 
indicators of, 5, 62 
longitudinal, 104 
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evaluator, 30, 32 
goals, 23 
objectives, 5, 62 

protocol, 39, 76, 104 

psychometrics, 4 



questions 

closed-ended, 37 
open-ended, 38, 106 



rating scale. 4, 98, 122 

reading 

proficiency, 44, 45 

recruitment 
staff, 26-27 
student, 26-27 

reliability, 1 1, 70 

s 

samples 

performance, 3, 21, 97-103 
writing, 45, 75, 98, 100 

scoring 

holistic, 25, 74 
primary trait, 75 

skills 

discrete, 2, 17 

Spanish, 28, 129 

staff 

bilingual, 27, 38 
knowledge, 9 

recruitment and '.raining, 27 

stakeholders, 27, 33. 94 

standardized 
assessment, 2 
measures, 62 
tests, 125-129 

standards 
curriculum, 19 



stimulation 
external, 17 

surveys, 3, 37-39, 94-97 
results, 42, 97 



test critiques, 128-129 

tests, standardized, 2, 41, 125- 
129 

Basic English Skills Test 

(BEST), 2, 129 
Comprehensive Adult Student 

Assessment System 

(CASAS), 2/l29 

time line, 27 

Title VII, 1, 10 

transiency, 1 1 

{see also attendance) 

u 

United States Refugee 
Resettlement Program, 40 

usage survey, 1 22 



validity, 1 1 
Vietnamese, 28, 38 

w 

workplace literacy, 129-131 

writing 
abilities, 75 

assessment, 99, 101-103 
samples, 45, 75,98, 100 
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Language in Education: Theory and Practice 

The Educational Resources Information Center (ERIC), which is sup- 
ported by the Office of Educational Research and Improvement of 
the U.S. Department of Education, is a nationwide system of infor- 
mation centers, each responsible for a given educational level or 
field of study. ERIC s basic objective is to make developments in 
educational research, instruction, and teacher training readily 
accessible to educators and members of related professions. 

The ERIC Clearinghouse on Languages and Linguistics (ERIC/CLL), one 

of the specialized information centers in the ERIC system, is oper- 
ated by the Center for Applied Linguistics (CAL) and is specifically 
responsible for the collection and dissemination of information on 
research in languages and linguistics and on the application of 
research to language teaching and learning. 

In 1989, CAL was awarded a contract to expand the activities of 
ERIC/CLL through the establishment of an adjunct ERIC clearing- 
house, the National Clearinghouse for ESL Literacy Education (NCLE)* 
NCLE s specific focus is literacy education for language minority 
adults and out-of-school youth. 

ERIC/CLL and NCLE commission recognized authorities in lan- 
guages, linguistics, adult literacy education, and English as a second 
language (ESL) to write about current issues in these fields. Mono- 
graphs, intended for educators, researchers, and others interested in 
language education, are published under the series title, Language in 
Education: Theory and Practice (LIE) .The UE series includes practi- 
cal guides for teachers, state-of-the-art papers, research reviews, and 
collected reports. 

For further information on the ERIC system, ERIC/CLL, or NCLE, 
contact either clearinghouse at the Center for Applied Linguistics, 
1118 22nd Street, NW, Washington, DC 20037. 

Joy Kreejl Peyton, Fran Keenan. Series Editors 
Vickie Lewelling, EklC/CLL Publicatiom Coordinator 
Miriam J. Burl SCLE Publications Coordinator 
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Other LIE Titles Available from Delta Systems 



The fallowing are other titles in the Language in Education series 
published by the Center for Applied Linguistics and Delta Systems 
Co., Inc.: 

Adult Biliterdcy in the United States (ISBN 0-937 354-83-X) 
edited by David Spener 

Approaches to Adult ESL Literacy Instruction (ISBN 0-93 73~>4»82*l) 
edited byJoAnn Crandall and Joy Kreeft Peyton 

Cooperative Learning: A Respoyjse to Linguistic and Cultural 
Diversity (ISBN 0-937354-81-3) 
edited by Daniel D. Holt 

Making Meaning, Making Change: Participatory Curriculum 
Development for Adult ESL Literacy (ISBN 0-93 7354- 79- 1 ) 
by Elsa Roberts Auerbach 

Speaking of Language: An Inter national Guide to Language 
Service Organizations (ISBN 0-937354 80-5) 

edited by Paula Conru, Vickie Lewelling, and Whitney Stewart 

Talking Shop: A Curriculum Sourcebook for Participatory Adult 
ESL (ISBN 0-93 7354-78-3) 

by Andrea Nash, Ann Cason, Madeline Rhum, Loren McGruil, and 
Rosario Gomez-Sanford 

To order any of these titles, call Delta Systems, Co., Inc. at 
(800) 323-8270 or (815) 363-3582 (9-5 EST) or write to them at 
1400 Miller Pkwy., McHenry, IL 60050. 
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ASSESSING SUCCESS IN 
FAMILY LITERACY PROJECTS: 
ALTERNATIVE APPROACHES TO 
ASSESSMENT AND EVALUATION 

Daniel D. Holt, Ed. 

Alternative assessment, unlike standardized 
tests, can be linked directly to the goals and 
curriculum of a literacy project,giving project 
staff immediate feedback for planning 
subsequent learning activities. With an 
emphasis on four alternative assessment 
approaches — surveys, interviews, observation 
measures, and performance samples — this 
handbook gives^^racy project staff working 
with language im^prify learners the means to 
gain accurate and comprehensive information 
about learners' progress and overall project 
effectiveness. 

Assessing Success provides many sample 
assessment tools that can be customized for 
use in literacy projects that serve language 
minority families. 





ISBN 0-93-7354-85-6 
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